powerdns authoritative 4.0.0 alpha stops working from time to time

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

powerdns authoritative 4.0.0 alpha stops working from time to time

Keresztes Péter-Zoltán
Hello, 

I have installed the new 4.0 version of the pdns. It works great however from time to time it suddenly stops working. In the logs I see lots of these:

Feb 26 07:42:34 dns1 pdns[6514]: Backend reported condition which prevented lookup (GSQLBackend lookup query: Attempt to bind more parameters than query has: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type=? and name=?) sending out servfail

Anything I can do about that?

regards,
Peter

_______________________________________________
Pdns-users mailing list
[hidden email]
http://mailman.powerdns.com/mailman/listinfo/pdns-users
Reply | Threaded
Open this post in threaded view
|

Re: powerdns authoritative 4.0.0 alpha stops working from time to time

Pieter Lexis-2
Hi Peter,

On Fri, 26 Feb 2016 14:49:16 +0200
Keresztes Péter-Zoltán <[hidden email]> wrote:

> I have installed the new 4.0 version of the pdns. It works great however from time to time it suddenly stops working. In the logs I see lots of these:

By "stops working", do you mean that the process dies or it just stops responding or it only sends out SERVFAILs?

> Feb 26 07:42:34 dns1 pdns[6514]: Backend reported condition which prevented lookup (GSQLBackend lookup query: Attempt to bind more parameters than query has: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type=? and name=?) sending out servfail

This is the `basic-query`, interesting that it gives this error, as we never bind more than 2 elements in the code.

If your server is not stressed for resources could you set `query-logging=yes` in the pdns.conf (with possibly `loglevel=7` as well if no queries show up in syslog) to identify the query that causes this and share the results with us?

Secondly, could you share your pdns.conf (without passwords)?

Best regards,

Pieter

--
Pieter Lexis
PowerDNS.COM BV -- https://www.powerdns.com
_______________________________________________
Pdns-users mailing list
[hidden email]
http://mailman.powerdns.com/mailman/listinfo/pdns-users
Reply | Threaded
Open this post in threaded view
|

Re: powerdns authoritative 4.0.0 alpha stops working from time to time

Keresztes Péter-Zoltán
I have enabled the query logging and increased the log level  however now I need to wait for it to happen which should happen in the next two days. That is the frequency this issue is happening.

By stop working I mean that the powerdns does not send response to the request.

The configuration file look like this:

cache-ttl=20
config-dir=/etc/powerdns
daemon=yes
default-soa-edit=ns1.dnsmadefree.com
default-soa-mail=[hidden email]
default-soa-name=ns1.dnsmadefree.com
default-ttl=3600
disable-tcp=no
distributor-threads=3
guardian=yes
include-dir=/etc/powerdns/pdns.d
launch=gmysql
local-address=0.0.0.0
local-port=53
loglevel=4
master=no
max-tcp-connections=10
module-dir=/usr/lib/x86_64-linux-gnu/pdns
negquery-cache-ttl=60
query-cache-ttl=20
receiver-threads=1
retrieval-threads=2
reuseport=yes
server-id=ns1.dnsmadefree.com
setgid=pdns
setuid=pdns
signing-threads=3
soa-expire-default=604800
soa-minimum-ttl=3600
soa-refresh-default=10800
soa-retry-default=3600
write-pid=yes

gmysql-dnssec
gmysql-user=someuser
gmysql-password=somepass
gmysql-dbname=somedb
gmysql-host=10.160.125.69

Regards,
Peter

> On 26 Feb 2016, at 15:18, Pieter Lexis <[hidden email]> wrote:
>
> Hi Peter,
>
> On Fri, 26 Feb 2016 14:49:16 +0200
> Keresztes Péter-Zoltán <[hidden email]> wrote:
>
>> I have installed the new 4.0 version of the pdns. It works great however from time to time it suddenly stops working. In the logs I see lots of these:
>
> By "stops working", do you mean that the process dies or it just stops responding or it only sends out SERVFAILs?
>
>> Feb 26 07:42:34 dns1 pdns[6514]: Backend reported condition which prevented lookup (GSQLBackend lookup query: Attempt to bind more parameters than query has: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type=? and name=?) sending out servfail
>
> This is the `basic-query`, interesting that it gives this error, as we never bind more than 2 elements in the code.
>
> If your server is not stressed for resources could you set `query-logging=yes` in the pdns.conf (with possibly `loglevel=7` as well if no queries show up in syslog) to identify the query that causes this and share the results with us?
>
> Secondly, could you share your pdns.conf (without passwords)?
>
> Best regards,
>
> Pieter
>
> --
> Pieter Lexis
> PowerDNS.COM BV -- https://www.powerdns.com
> _______________________________________________
> Pdns-users mailing list
> [hidden email]
> http://mailman.powerdns.com/mailman/listinfo/pdns-users

_______________________________________________
Pdns-users mailing list
[hidden email]
http://mailman.powerdns.com/mailman/listinfo/pdns-users
Reply | Threaded
Open this post in threaded view
|

Re: powerdns authoritative 4.0.0 alpha stops working from time to time

Peter van Dijk
In reply to this post by Pieter Lexis-2
Hello,

On 26 Feb 2016, at 14:18, Pieter Lexis wrote:

> This is the `basic-query`, interesting that it gives this error, as we
> never bind more than 2 elements in the code.

Unless there is a code path in which we never reach

   try {
     d_query_stmt->reset();
   } catch (SSqlException &e) {
    ..
    ..

inside GSQLBackend::get

Kind regards,
--
Peter van Dijk
PowerDNS.COM BV - https://www.powerdns.com/
_______________________________________________
Pdns-users mailing list
[hidden email]
http://mailman.powerdns.com/mailman/listinfo/pdns-users
Reply | Threaded
Open this post in threaded view
|

Re: powerdns authoritative 4.0.0 alpha stops working from time to time

Ruben d'Arco
Just a "fyi" i've expirienced the same behaviour a while ago.
That was on a Ubuntu 14.04LTS system. PDNS works fine after restarting, but i haven't been able to reproduce the issue.
My config is like this:

allow-dnsupdate-from=127.0.0.0/8,192.168.0.100/32
any-to-tcp=yes
config-dir=/etc/powerdns
daemon=yes
dnsupdate=yes
guardian=yes
launch=gmysql
gmysql-user=pdns
gmysql-host=localhost
gmysql-password=p0w3rdns
gmysql-dbname=pdns
local-address=127.0.0.1
loglevel=9999
setgid=pdns
setuid=pdns


Regards,
        Ruben


On Fri, Feb 26, 2016 at 09:39:55PM +0100, Peter van Dijk wrote:

> Hello,
>
> On 26 Feb 2016, at 14:18, Pieter Lexis wrote:
>
> >This is the `basic-query`, interesting that it gives this error, as we
> >never bind more than 2 elements in the code.
>
> Unless there is a code path in which we never reach
>
>   try {
>     d_query_stmt->reset();
>   } catch (SSqlException &e) {
>    ..
>    ..
>
> inside GSQLBackend::get
>
> Kind regards,
> --
> Peter van Dijk
> PowerDNS.COM BV - https://www.powerdns.com/
> _______________________________________________
> Pdns-users mailing list
> [hidden email]
> http://mailman.powerdns.com/mailman/listinfo/pdns-users
_______________________________________________
Pdns-users mailing list
[hidden email]
http://mailman.powerdns.com/mailman/listinfo/pdns-users
Reply | Threaded
Open this post in threaded view
|

Re: powerdns authoritative 4.0.0 alpha stops working from time to time

Keresztes Péter-Zoltán
In reply to this post by Peter van Dijk
Hello,

I finally have some logs when this issue is happening: 

Here is what I can find in the logs:

Mar  5 16:57:21 dns1 pdns[7111]: Backend reported condition which prevented lookup (GSQLBackend lookup query: Attempt to bind more parameters than query has: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type=? and name=?) sending out servfail
Mar  5 16:57:21 dns1 pdns[7111]: Backend reported condition which prevented lookup (GSQLBackend lookup query: Attempt to bind more parameters than query has: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type=? and name=?) sending out servfail
Mar  5 16:57:21 dns1 pdns_server[7109]: Mar 05 16:57:21 Backend reported condition which prevented lookup (GSQLBackend lookup query: Attempt to bind more parameters than query has: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type=? and name=?) sending out servfail
Mar  5 16:57:21 dns1 pdns_server[7109]: Mar 05 16:57:21 Backend reported condition which prevented lookup (GSQLBackend lookup query: Attempt to bind more parameters than query has: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type=? and name=?) sending out servfail

Regards,
Peter

On 26 Feb 2016, at 22:39, Peter van Dijk <[hidden email]> wrote:

Hello,

On 26 Feb 2016, at 14:18, Pieter Lexis wrote:

This is the `basic-query`, interesting that it gives this error, as we never bind more than 2 elements in the code.

Unless there is a code path in which we never reach

 try {
   d_query_stmt->reset();
 } catch (SSqlException &e) {
  ..
  ..

inside GSQLBackend::get

Kind regards,
--
Peter van Dijk
PowerDNS.COM BV - https://www.powerdns.com/
_______________________________________________
Pdns-users mailing list
[hidden email]
http://mailman.powerdns.com/mailman/listinfo/pdns-users


_______________________________________________
Pdns-users mailing list
[hidden email]
http://mailman.powerdns.com/mailman/listinfo/pdns-users
Reply | Threaded
Open this post in threaded view
|

Re: powerdns authoritative 4.0.0 alpha stops working from time to time

Keresztes Péter-Zoltán
In reply to this post by Keresztes Péter-Zoltán
Hello,

I finally have some logs when this issue is happening: 

Here is what I can find in the logs:

Mar  5 16:57:21 dns1 pdns[7111]: Backend reported condition which prevented lookup (GSQLBackend lookup query: Attempt to bind more parameters than query has: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type=? and name=?) sending out servfail
Mar  5 16:57:21 dns1 pdns[7111]: Backend reported condition which prevented lookup (GSQLBackend lookup query: Attempt to bind more parameters than query has: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type=? and name=?) sending out servfail
Mar  5 16:57:21 dns1 pdns_server[7109]: Mar 05 16:57:21 Backend reported condition which prevented lookup (GSQLBackend lookup query: Attempt to bind more parameters than query has: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type=? and name=?) sending out servfail
Mar  5 16:57:21 dns1 pdns_server[7109]: Mar 05 16:57:21 Backend reported condition which prevented lookup (GSQLBackend lookup query: Attempt to bind more parameters than query has: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type=? and name=?) sending out servfail

Regards,
Peter

On 26 Feb 2016, at 20:31, Keresztes Péter-Zoltán <[hidden email]> wrote:

I have enabled the query logging and increased the log level  however now I need to wait for it to happen which should happen in the next two days. That is the frequency this issue is happening.

By stop working I mean that the powerdns does not send response to the request.

The configuration file look like this:

cache-ttl=20
config-dir=/etc/powerdns
daemon=yes
default-soa-edit=ns1.dnsmadefree.com
[hidden email]
default-soa-name=ns1.dnsmadefree.com
default-ttl=3600
disable-tcp=no
distributor-threads=3
guardian=yes
include-dir=/etc/powerdns/pdns.d
launch=gmysql
local-address=0.0.0.0
local-port=53
loglevel=4
master=no
max-tcp-connections=10
module-dir=/usr/lib/x86_64-linux-gnu/pdns
negquery-cache-ttl=60
query-cache-ttl=20
receiver-threads=1
retrieval-threads=2
reuseport=yes
server-id=ns1.dnsmadefree.com
setgid=pdns
setuid=pdns
signing-threads=3
soa-expire-default=604800
soa-minimum-ttl=3600
soa-refresh-default=10800
soa-retry-default=3600
write-pid=yes

gmysql-dnssec
gmysql-user=someuser
gmysql-password=somepass
gmysql-dbname=somedb
gmysql-host=10.160.125.69

Regards,
Peter
On 26 Feb 2016, at 15:18, Pieter Lexis <[hidden email]> wrote:

Hi Peter,

On Fri, 26 Feb 2016 14:49:16 +0200
Keresztes Péter-Zoltán <[hidden email]> wrote:

I have installed the new 4.0 version of the pdns. It works great however from time to time it suddenly stops working. In the logs I see lots of these:

By "stops working", do you mean that the process dies or it just stops responding or it only sends out SERVFAILs?

Feb 26 07:42:34 dns1 pdns[6514]: Backend reported condition which prevented lookup (GSQLBackend lookup query: Attempt to bind more parameters than query has: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type=? and name=?) sending out servfail

This is the `basic-query`, interesting that it gives this error, as we never bind more than 2 elements in the code.

If your server is not stressed for resources could you set `query-logging=yes` in the pdns.conf (with possibly `loglevel=7` as well if no queries show up in syslog) to identify the query that causes this and share the results with us?

Secondly, could you share your pdns.conf (without passwords)?

Best regards,

Pieter

--
Pieter Lexis
PowerDNS.COM BV -- https://www.powerdns.com
_______________________________________________
Pdns-users mailing list
[hidden email]
http://mailman.powerdns.com/mailman/listinfo/pdns-users



_______________________________________________
Pdns-users mailing list
[hidden email]
http://mailman.powerdns.com/mailman/listinfo/pdns-users
Reply | Threaded
Open this post in threaded view
|

Re: powerdns authoritative 4.0.0 alpha stops working from time to time

Radosław Piliszek
In reply to this post by Keresztes Péter-Zoltán
Hello,

I am having the same problem using PowerDNS Authoritative Server
4.0.0-alpha2 from CentOS 7 repository and the MySQL backend.

I enabled the highest level of logging (9).

And here are the results...

First failed queries (e.g. three times for three connections to the backend):

Remote 127.0.0.1 wants '...', do = 0, bufsize = 512: packetcache MISS
Query: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM
records WHERE disabled=0 and type=? and name=?
Backend reported condition which prevented lookup (GSQLBackend lookup
query: Could not execute mysql statement: SELECT
content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE
disabled=0 and type=? and name=?: Lost connection to MySQL server
during query) sending out servfail

All following queries:

Remote 127.0.0.1 wants '...', do = 0, bufsize = 512: packetcache MISS
Backend reported condition which prevented lookup (GSQLBackend lookup
query: Attempt to bind more parameters than query has: SELECT
content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE
disabled=0 and type=? and name=?) sending out servfail

There are no other log messages in between these.

After the connection is lost, the "Attempt to bind more parameters
than query has" (as already mentioned here) appears until the PowerDNS
server is restarted...

Also, in my setup the connection is lost due to timeout because in
this test environment queries appear very rarely. I think PowerDNS
should handle timeouted sessions more gracefully like trying to
restart the connection instead of failing the DNS request...

In summary, two problems:
1) Connection to backend lost = queries stuck on error.
2) Session timeouts are not handled gracefully.

Kind regards,
Radosław Piliszek

> On 26 Feb 2016, at 15:18, Pieter Lexis <pieter.lexis at powerdns.com> wrote:
>
> Hi Peter,
>
> On Fri, 26 Feb 2016 14:49:16 +0200
> Keresztes Péter-Zoltán <zozo at z0z0.tk> wrote:
>
>> I have installed the new 4.0 version of the pdns. It works great however from time to time it suddenly stops working. In the logs I see lots of these:
>
> By "stops working", do you mean that the process dies or it just stops responding or it only sends out SERVFAILs?
>
>> Feb 26 07:42:34 dns1 pdns[6514]: Backend reported condition which prevented lookup (GSQLBackend lookup query: Attempt to bind more parameters than query has: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type=? and name=?) sending out servfail
>
> This is the `basic-query`, interesting that it gives this error, as we never bind more than 2 elements in the code.
>
> If your server is not stressed for resources could you set `query-logging=yes` in the pdns.conf (with possibly `loglevel=7` as well if no queries show up in syslog) to identify the query that causes this and share the results with us?
>
> Secondly, could you share your pdns.conf (without passwords)?
>
> Best regards,
>
> Pieter
>
> --
> Pieter Lexis
> PowerDNS.COM BV -- https://www.powerdns.com
> _______________________________________________
> Pdns-users mailing list
> Pdns-users at mailman.powerdns.com
> http://mailman.powerdns.com/mailman/listinfo/pdns-users
_______________________________________________
Pdns-users mailing list
[hidden email]
http://mailman.powerdns.com/mailman/listinfo/pdns-users
Reply | Threaded
Open this post in threaded view
|

Re: powerdns authoritative 4.0.0 alpha stops working from time to time

shthead
In reply to this post by Keresztes Péter-Zoltán
Hi,

I too have ran into this issue today. I can reproduce it easily.

Running 4.0.0~alpha2-1pdns.jessie with the MySQL backend. I restart the
MySQL server. After it is done restarting any queries to PowerDNS return
a SERVFAIL with this line logged first:

Backend reported condition which prevented lookup (GSQLBackend lookup
query: Could not execute mysql statement: SELECT
content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE
disabled=0 and type=? and name=?: Lost connection to MySQL server during
query) sending out servfail

Subsequent queries then log:

Backend reported condition which prevented lookup (GSQLBackend lookup
query: Attempt to bind more parameters than query has: SELECT
content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE
disabled=0 and type=? and name=?) sending out servfail

Restarting PowerDNS fixes it.

Thanks
_______________________________________________
Pdns-users mailing list
[hidden email]
http://mailman.powerdns.com/mailman/listinfo/pdns-users