An important update on new PowerDNS Products

classic Classic list List threaded Threaded
11 messages Options
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

An important update on new PowerDNS Products

bert hubert-3

Hi everybody,

This is a heads-up on some announcements you will be seeing on powerdns.com relating to new PowerDNS products which (gasp) are not fully Open Source. We know this is a sensitive subject, so before we go live, we want to inform you fully of what we are doing. We’d also like to hear & incorporate your feedback.

The tl;dr: PowerDNS will remain enthusiastically Open Source, but we will be selling a ready-to use ‘Platform’ of PowerDNS Open Source & other technologies, without degrading our current products. For details, please read on.

As you may know, PowerDNS sells support on the core nameserver technologies: PowerDNS Authoritative ServerPowerDNS Recursor and dnsdist. And this is going well, well enough to fund four full-time developers & engineers. This delivers a lot of value to the Open Source world.

Over the past few years, as part of our paid support, we have also been delivering custom PowerDNS configurations based on our open source products. Such configurations integrate with Graphite, Ansible, exabgp, bird, iptables and loads of other products to deliver features like parental control, configuration management, governmental/judicial blacklists, DoS protection of (legacy) nameservers, malware filtering, quarantining, NXDOMAIN redirection, “customer communications”, monitoring, user-experience graphing, audit trail of configuration changes, (management) reporting, webbased control, BGP/OSPF/VRRP failover, ‘production’ DNS64 etc etc.

What we have also found is that many of our users (big hosters, large scale telecommunications service providers) need more from us than “/usr/sbin/pdns_recursor”. Although PowerDNS can easily be integrated with lots of things to deliver powerful functionalities and many of our users still love open source, they would prefer to get it packaged in a more ready to use way.

Putting it more strongly: we have learned that many organizations simply no longer have the time or desire to assemble all the technologies themselves around our Open Source products.

We will therefore be marketing the additional functionalities we have been delivering to our customers as a product tentatively called the “PowerDNS Platform”. I say tentatively because we want to inform you of this news first, even before we have settled on a name and updated our website with the new product.

The “PowerDNS Platform” as we ship it consists of our core unmodified Open Source products, plus loads of other open source technologies, combined with a management shell that is not an Open Source product that we’ll in fact sell.

Now, we understand this may be worrying some to some of you. Some formerly truly Open Source products like MySQL are going down a path where you can see their products turning into a sales pitch for the commercially licensed version. Some other Open Source nameservers have used their liberal licensing to sell ‘subscriber versions‘ of their software that have additional core functionalities. This might create doubt if the product in its Open Source version will retain the capabilities discerning users of open software demand.

We would therefore like to clarify that we regard our core Open Source products as our crown jewels, jewels which only shine because we are an integral part of the DNS and PowerDNS Communities with whom we work together to create great software.We will continue to make sure that our nameserver software is a viable and hopefully even the best choice for the Internet at large. And in fact, there will not be “two versions” of the PowerDNS nameserver software: of the actual daemons there will be just one version – also because we would otherwise not get the advantages of scale we get from over 150000 deployments!

Simultaneously, we hope that by bringing PowerDNS in a more integrated fashion will enable more companies to benefit from running Open Source & open standards based software. Because this is what deeply believe in – that the future of the world is open, and that software can simultaneously be good Open Source and also work well in a commercial environment.

Thank you for reading this to the end! We would like to hear your feedback and perhaps worries. Please contact me on[hidden email] to let us know your thoughts and concerns.

Bert


_______________________________________________
Pdns-users mailing list
[hidden email]
http://mailman.powerdns.com/mailman/listinfo/pdns-users
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Pdns-dev] An important update on new PowerDNS Products

Veaceslav Mindru
Meh one more turned the dark side ...

VM

On Tue, Feb 23, 2016 at 11:14 AM, bert hubert <[hidden email]> wrote:

> Hi everybody,
>
> This is a heads-up on some announcements you will be seeing on powerdns.com
> relating to new PowerDNS products which (gasp) are not fully Open Source. We
> know this is a sensitive subject, so before we go live, we want to inform
> you fully of what we are doing. We’d also like to hear & incorporate your
> feedback.
>
> The tl;dr: PowerDNS will remain enthusiastically Open Source, but we will be
> selling a ready-to use ‘Platform’ of PowerDNS Open Source & other
> technologies, without degrading our current products. For details, please
> read on.
>
> As you may know, PowerDNS sells support on the core nameserver technologies:
> PowerDNS Authoritative Server, PowerDNS Recursor and dnsdist. And this is
> going well, well enough to fund four full-time developers & engineers. This
> delivers a lot of value to the Open Source world.
>
> Over the past few years, as part of our paid support, we have also been
> delivering custom PowerDNS configurations based on our open source products.
> Such configurations integrate with Graphite, Ansible, exabgp, bird, iptables
> and loads of other products to deliver features like parental control,
> configuration management, governmental/judicial blacklists, DoS protection
> of (legacy) nameservers, malware filtering, quarantining, NXDOMAIN
> redirection, “customer communications”, monitoring, user-experience
> graphing, audit trail of configuration changes, (management) reporting,
> webbased control, BGP/OSPF/VRRP failover, ‘production’ DNS64 etc etc.
>
> What we have also found is that many of our users (big hosters, large scale
> telecommunications service providers) need more from us than
> “/usr/sbin/pdns_recursor”. Although PowerDNS can easily be integrated with
> lots of things to deliver powerful functionalities and many of our users
> still love open source, they would prefer to get it packaged in a more ready
> to use way.
>
> Putting it more strongly: we have learned that many organizations simply no
> longer have the time or desire to assemble all the technologies themselves
> around our Open Source products.
>
> We will therefore be marketing the additional functionalities we have been
> delivering to our customers as a product tentatively called the “PowerDNS
> Platform”. I say tentatively because we want to inform you of this news
> first, even before we have settled on a name and updated our website with
> the new product.
>
> The “PowerDNS Platform” as we ship it consists of our core unmodified Open
> Source products, plus loads of other open source technologies, combined with
> a management shell that is not an Open Source product that we’ll in fact
> sell.
>
> Now, we understand this may be worrying some to some of you. Some formerly
> truly Open Source products like MySQL are going down a path where you can
> see their products turning into a sales pitch for the commercially licensed
> version. Some other Open Source nameservers have used their liberal
> licensing to sell ‘subscriber versions‘ of their software that have
> additional core functionalities. This might create doubt if the product in
> its Open Source version will retain the capabilities discerning users of
> open software demand.
>
> We would therefore like to clarify that we regard our core Open Source
> products as our crown jewels, jewels which only shine because we are an
> integral part of the DNS and PowerDNS Communities with whom we work together
> to create great software.We will continue to make sure that our nameserver
> software is a viable and hopefully even the best choice for the Internet at
> large. And in fact, there will not be “two versions” of the PowerDNS
> nameserver software: of the actual daemons there will be just one version –
> also because we would otherwise not get the advantages of scale we get from
> over 150000 deployments!
>
> Simultaneously, we hope that by bringing PowerDNS in a more integrated
> fashion will enable more companies to benefit from running Open Source &
> open standards based software. Because this is what deeply believe in – that
> the future of the world is open, and that software can simultaneously be
> good Open Source and also work well in a commercial environment.
>
> Thank you for reading this to the end! We would like to hear your feedback
> and perhaps worries. Please contact me [hidden email] to let us
> know your thoughts and concerns.
>
> Bert
>
>
> _______________________________________________
> Pdns-dev mailing list
> [hidden email]
> http://mailman.powerdns.com/mailman/listinfo/pdns-dev
>



--
Veaceslav Mindru
PNT Devops Sysops
_______________________________________________
Pdns-users mailing list
[hidden email]
http://mailman.powerdns.com/mailman/listinfo/pdns-users
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Pdns-dev] An important update on new PowerDNS Products

Martijn Grendelman
Says someone using a redhat.com email address. LMAO.

Martijn.



Op 23-2-2016 om 11:19 schreef Veaceslav Mindru:

> Meh one more turned the dark side ...
>
> VM
>
> On Tue, Feb 23, 2016 at 11:14 AM, bert hubert <[hidden email]> wrote:
>> Hi everybody,
>>
>> This is a heads-up on some announcements you will be seeing on powerdns.com
>> relating to new PowerDNS products which (gasp) are not fully Open Source. We
>> know this is a sensitive subject, so before we go live, we want to inform
>> you fully of what we are doing. We’d also like to hear & incorporate your
>> feedback.
>>
>> The tl;dr: PowerDNS will remain enthusiastically Open Source, but we will be
>> selling a ready-to use ‘Platform’ of PowerDNS Open Source & other
>> technologies, without degrading our current products. For details, please
>> read on.
>>
>> As you may know, PowerDNS sells support on the core nameserver technologies:
>> PowerDNS Authoritative Server, PowerDNS Recursor and dnsdist. And this is
>> going well, well enough to fund four full-time developers & engineers. This
>> delivers a lot of value to the Open Source world.
>>
>> Over the past few years, as part of our paid support, we have also been
>> delivering custom PowerDNS configurations based on our open source products.
>> Such configurations integrate with Graphite, Ansible, exabgp, bird, iptables
>> and loads of other products to deliver features like parental control,
>> configuration management, governmental/judicial blacklists, DoS protection
>> of (legacy) nameservers, malware filtering, quarantining, NXDOMAIN
>> redirection, “customer communications”, monitoring, user-experience
>> graphing, audit trail of configuration changes, (management) reporting,
>> webbased control, BGP/OSPF/VRRP failover, ‘production’ DNS64 etc etc.
>>
>> What we have also found is that many of our users (big hosters, large scale
>> telecommunications service providers) need more from us than
>> “/usr/sbin/pdns_recursor”. Although PowerDNS can easily be integrated with
>> lots of things to deliver powerful functionalities and many of our users
>> still love open source, they would prefer to get it packaged in a more ready
>> to use way.
>>
>> Putting it more strongly: we have learned that many organizations simply no
>> longer have the time or desire to assemble all the technologies themselves
>> around our Open Source products.
>>
>> We will therefore be marketing the additional functionalities we have been
>> delivering to our customers as a product tentatively called the “PowerDNS
>> Platform”. I say tentatively because we want to inform you of this news
>> first, even before we have settled on a name and updated our website with
>> the new product.
>>
>> The “PowerDNS Platform” as we ship it consists of our core unmodified Open
>> Source products, plus loads of other open source technologies, combined with
>> a management shell that is not an Open Source product that we’ll in fact
>> sell.
>>
>> Now, we understand this may be worrying some to some of you. Some formerly
>> truly Open Source products like MySQL are going down a path where you can
>> see their products turning into a sales pitch for the commercially licensed
>> version. Some other Open Source nameservers have used their liberal
>> licensing to sell ‘subscriber versions‘ of their software that have
>> additional core functionalities. This might create doubt if the product in
>> its Open Source version will retain the capabilities discerning users of
>> open software demand.
>>
>> We would therefore like to clarify that we regard our core Open Source
>> products as our crown jewels, jewels which only shine because we are an
>> integral part of the DNS and PowerDNS Communities with whom we work together
>> to create great software.We will continue to make sure that our nameserver
>> software is a viable and hopefully even the best choice for the Internet at
>> large. And in fact, there will not be “two versions” of the PowerDNS
>> nameserver software: of the actual daemons there will be just one version –
>> also because we would otherwise not get the advantages of scale we get from
>> over 150000 deployments!
>>
>> Simultaneously, we hope that by bringing PowerDNS in a more integrated
>> fashion will enable more companies to benefit from running Open Source &
>> open standards based software. Because this is what deeply believe in – that
>> the future of the world is open, and that software can simultaneously be
>> good Open Source and also work well in a commercial environment.
>>
>> Thank you for reading this to the end! We would like to hear your feedback
>> and perhaps worries. Please contact me [hidden email] to let us
>> know your thoughts and concerns.
>>
>> Bert
>>
>>
>> _______________________________________________
>> Pdns-dev mailing list
>> [hidden email]
>> http://mailman.powerdns.com/mailman/listinfo/pdns-dev
>>
>
>
>
_______________________________________________
Pdns-users mailing list
[hidden email]
http://mailman.powerdns.com/mailman/listinfo/pdns-users
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Pdns-dev] An important update on new PowerDNS Products

Jan-Piet Mens-3
In reply to this post by Veaceslav Mindru
> Meh one more turned the dark side ...

Did you read the tl;dr?

> The tl;dr: PowerDNS will remain enthusiastically Open Source

        -JP
_______________________________________________
Pdns-users mailing list
[hidden email]
http://mailman.powerdns.com/mailman/listinfo/pdns-users
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Pdns-dev] An important update on new PowerDNS Products

bert hubert-3
In reply to this post by bert hubert-3
On Tue, Feb 23, 2016 at 01:27:57PM +0000, Justin Clift wrote:
>  > We would therefore like to clarify that we regard our core Open Source
> >  products as our crown jewels
>
> This seems to be missing the reason why the new "Platform" is better off
> non-OSS?

Hi Justin,

That is because it is not universally better. I think you personally might
find the "Platform" to be worse even!  But read on for why.

The best explanation may be found in the second to last link in the blogpost
or email (if you have HTML email) on
http://venturebeat.com/2015/12/06/its-actually-open-source-software-thats-eating-the-world/

About successful open source businesses, it says:

"They’ve figured out that customers are more than happy to pay for an
enterprise-grade version of the complete product, which may have security,
management, or integration enhancements and come with support. And they also
understand that keeping this type of functionality proprietary won’t
alienate the community supporting the project the way something such as a
performance enhancement would."

Basically everything in the Platform is something you could build/hook
yourself.  You and I personally could do that easily, and the companies
where we work would probably prefer that.  In business this is referred as
'best of breed approach'.  http://c2.com/cgi/wiki?BestOfBreed calls it:

".. the strategy of selecting the best product of each type
(and integrating them yourself), rather than selecting one large integrated
solution from a single vendor.".

Not every company is like that though: they may prefer something that comes
with loads of integrated features (graphing, failover, monitoring, graphical
user interface, configuration audit trail etc) out of the box.

Ideally, our "Platform" would consist of features that most of our Open
Source friends would prefer to assemble themselves using the large amount of
hooks (Pipe, Remotebackend, our backend API, our RestFUL/JSON API, Lua
support, Carbon output, dnsdist DNSTAP (near future)) that we offer for that.

The problem we are solving here is making ourselves more accessible for
those places that do not in fact want to assemble all this themselves -
without closing down the very open PowerDNS software.

Does this help?

        Bert

>
> Is it a case of wanting the Platform to be a competative differentiator?
> (GitHub style?)
>
> + Justin
_______________________________________________
Pdns-users mailing list
[hidden email]
http://mailman.powerdns.com/mailman/listinfo/pdns-users
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: An important update on new PowerDNS Products

Ciro Iriarte
In reply to this post by bert hubert-3
Out of curiosity, how does this part of the platform work?:

Long term full query logging & rapid searching
  • Dimensioned at a trillion queries/day (1000 billion) on commodity hardware with long term retention
  • For security research, lawful intercept/data retention requirements, customer intelligence, quality assurance/diagnostics

Reporting is still on my ToDo. Any blog post pending about that? :)

Regards,
CI.-

2016-02-23 7:14 GMT-03:00 bert hubert <[hidden email]>:

Hi everybody,

This is a heads-up on some announcements you will be seeing on powerdns.com relating to new PowerDNS products which (gasp) are not fully Open Source. We know this is a sensitive subject, so before we go live, we want to inform you fully of what we are doing. We’d also like to hear & incorporate your feedback.

The tl;dr: PowerDNS will remain enthusiastically Open Source, but we will be selling a ready-to use ‘Platform’ of PowerDNS Open Source & other technologies, without degrading our current products. For details, please read on.

As you may know, PowerDNS sells support on the core nameserver technologies: PowerDNS Authoritative ServerPowerDNS Recursor and dnsdist. And this is going well, well enough to fund four full-time developers & engineers. This delivers a lot of value to the Open Source world.

Over the past few years, as part of our paid support, we have also been delivering custom PowerDNS configurations based on our open source products. Such configurations integrate with Graphite, Ansible, exabgp, bird, iptables and loads of other products to deliver features like parental control, configuration management, governmental/judicial blacklists, DoS protection of (legacy) nameservers, malware filtering, quarantining, NXDOMAIN redirection, “customer communications”, monitoring, user-experience graphing, audit trail of configuration changes, (management) reporting, webbased control, BGP/OSPF/VRRP failover, ‘production’ DNS64 etc etc.

What we have also found is that many of our users (big hosters, large scale telecommunications service providers) need more from us than “/usr/sbin/pdns_recursor”. Although PowerDNS can easily be integrated with lots of things to deliver powerful functionalities and many of our users still love open source, they would prefer to get it packaged in a more ready to use way.

Putting it more strongly: we have learned that many organizations simply no longer have the time or desire to assemble all the technologies themselves around our Open Source products.

We will therefore be marketing the additional functionalities we have been delivering to our customers as a product tentatively called the “PowerDNS Platform”. I say tentatively because we want to inform you of this news first, even before we have settled on a name and updated our website with the new product.

The “PowerDNS Platform” as we ship it consists of our core unmodified Open Source products, plus loads of other open source technologies, combined with a management shell that is not an Open Source product that we’ll in fact sell.

Now, we understand this may be worrying some to some of you. Some formerly truly Open Source products like MySQL are going down a path where you can see their products turning into a sales pitch for the commercially licensed version. Some other Open Source nameservers have used their liberal licensing to sell ‘subscriber versions‘ of their software that have additional core functionalities. This might create doubt if the product in its Open Source version will retain the capabilities discerning users of open software demand.

We would therefore like to clarify that we regard our core Open Source products as our crown jewels, jewels which only shine because we are an integral part of the DNS and PowerDNS Communities with whom we work together to create great software.We will continue to make sure that our nameserver software is a viable and hopefully even the best choice for the Internet at large. And in fact, there will not be “two versions” of the PowerDNS nameserver software: of the actual daemons there will be just one version – also because we would otherwise not get the advantages of scale we get from over 150000 deployments!

Simultaneously, we hope that by bringing PowerDNS in a more integrated fashion will enable more companies to benefit from running Open Source & open standards based software. Because this is what deeply believe in – that the future of the world is open, and that software can simultaneously be good Open Source and also work well in a commercial environment.

Thank you for reading this to the end! We would like to hear your feedback and perhaps worries. Please contact me on[hidden email] to let us know your thoughts and concerns.

Bert


_______________________________________________
Pdns-users mailing list
[hidden email]
http://mailman.powerdns.com/mailman/listinfo/pdns-users




--
Ciro Iriarte
http://iriarte.it
--

_______________________________________________
Pdns-users mailing list
[hidden email]
https://mailman.powerdns.com/mailman/listinfo/pdns-users
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: An important update on new PowerDNS Products

bert hubert-3
On Fri, May 13, 2016 at 07:59:57PM -0400, Ciro Iriarte wrote:
> Out of curiosity, how does this part of the platform work?:

Hi Ciro,

In general, I don't want to spam the pdns-users people with information
about the PowerDNS Platform, as outlined on
https://www.powerdns.com/platform.html since it is not part of the open
source stuff, so most of the community won't have any use for the
information.

(the story of our non-open source work is on
https://blog.powerdns.com/2016/02/23/an-important-update-on-new-powerdns-products/
)

But let me explain how this hangs together and what is part of the open
source version. The PowerDNS Recursor 4.0.0 and dnsdist 1.0.0 have the
ability to emit a stream of protobufs over TCP/IP. In case of the Recursor,
this also has the 'policy reason' why a request was intercepted by the RPZ
module.

To receive that stream, use something like xinetd to listen on a TCP/IP port
and store the data to a file. It can then be processed by any tool that can
understand Protobuf. The schema is here:
https://github.com/PowerDNS/pdns/blob/master/pdns/dnsmessage.proto

In the very near future this will also be able to emit standard dnstap
messages.

> Long term full query logging & rapid searching
>
>    - Dimensioned at a trillion queries/day (1000 billion) on commodity
>    hardware with long term retention
>    - For security research, lawful intercept/data retention requirements,
>    customer intelligence, quality assurance/diagnostics

This describes our protobuf receiver 'dstore' which through some clever
programming techniques can store trillions of DNS messages and serve them up
again reasonably quickly. It is not a generic database, but it is really
fast and nearly maintenance free and has no further dependencies (so you
don't need to be a "big data engineer" to benefit from it).

This can be very useful to investigate customer complaints of DNS slowness,
or that a domain was down etc. It is also extremely powerful for finding
infected users. A commandline like:

$ dgrep t=week pr=spamhaus-dbl | jq ".items[].origRequestor"  | sort | uniq -c \
| sort -rn | head -10

.. will find in a few seconds the top-10 IP addresses that over the past week
had the most queries  blocked by the 'spamhaus-dbl' RPZ. The output of dgrep is JSON,
easily queried and selected by jq.

But again - I don't want to promote our commercial Platform offering here
too much.  For the open source world, you should be able to bake up a
solution based on elastic search, kibana etc that ingests our protobufs.


        Bert

_______________________________________________
Pdns-users mailing list
[hidden email]
https://mailman.powerdns.com/mailman/listinfo/pdns-users
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: An important update on new PowerDNS Products

Gavin Henry


>
> This describes our protobuf receiver 'dstore' which through some clever
> programming techniques can store trillions of DNS messages and serve them up
> again reasonably quickly. It is not a generic database, but it is really
> fast and nearly maintenance free and has no further dependencies (so you
> don't need to be a "big data engineer" to benefit from it).
>

This wouldn't be lmdb would it?

Thanks.


_______________________________________________
Pdns-users mailing list
[hidden email]
https://mailman.powerdns.com/mailman/listinfo/pdns-users
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: An important update on new PowerDNS Products

bert hubert-3
On Sat, May 14, 2016 at 02:39:38PM +0100, Gavin Henry wrote:
> > again reasonably quickly. It is not a generic database, but it is really
> > fast and nearly maintenance free and has no further dependencies (so you
> > don't need to be a "big data engineer" to benefit from it).
> >
>
> This wouldn't be lmdb would it?

No, nothing as generically powerful as that. The goal for dstore is to be
absolutely robust even with 200TB+ size databases. Recovering an index takes
days at that size. But, for further questions, please do not use
pdns-users. We might launch pdns-platform-announce for people that really
want to know though!

        Bert
_______________________________________________
Pdns-users mailing list
[hidden email]
https://mailman.powerdns.com/mailman/listinfo/pdns-users
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: An important update on new PowerDNS Products

Gavin Henry


> No, nothing as generically powerful as that. The goal for dstore is to be
> absolutely robust even with 200TB+ size databases. Recovering an index takes
> days at that size. But, for further questions, please do not use
> pdns-users. We might launch pdns-platform-announce for people that really
> want to know though!
>

Sounds interesting! (lmdb doesn't crash)


_______________________________________________
Pdns-users mailing list
[hidden email]
https://mailman.powerdns.com/mailman/listinfo/pdns-users
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: An important update on new PowerDNS Products

Ciro Iriarte
In reply to this post by bert hubert-3
Interesting!, thanks a lot Bert.

Regards,
CI.-

2016-05-14 9:10 GMT-04:00 bert hubert <[hidden email]>:
On Fri, May 13, 2016 at 07:59:57PM -0400, Ciro Iriarte wrote:
> Out of curiosity, how does this part of the platform work?:

Hi Ciro,

In general, I don't want to spam the pdns-users people with information
about the PowerDNS Platform, as outlined on
https://www.powerdns.com/platform.html since it is not part of the open
source stuff, so most of the community won't have any use for the
information.

(the story of our non-open source work is on
https://blog.powerdns.com/2016/02/23/an-important-update-on-new-powerdns-products/
)

But let me explain how this hangs together and what is part of the open
source version. The PowerDNS Recursor 4.0.0 and dnsdist 1.0.0 have the
ability to emit a stream of protobufs over TCP/IP. In case of the Recursor,
this also has the 'policy reason' why a request was intercepted by the RPZ
module.

To receive that stream, use something like xinetd to listen on a TCP/IP port
and store the data to a file. It can then be processed by any tool that can
understand Protobuf. The schema is here:
https://github.com/PowerDNS/pdns/blob/master/pdns/dnsmessage.proto

In the very near future this will also be able to emit standard dnstap
messages.

> Long term full query logging & rapid searching
>
>    - Dimensioned at a trillion queries/day (1000 billion) on commodity
>    hardware with long term retention
>    - For security research, lawful intercept/data retention requirements,
>    customer intelligence, quality assurance/diagnostics

This describes our protobuf receiver 'dstore' which through some clever
programming techniques can store trillions of DNS messages and serve them up
again reasonably quickly. It is not a generic database, but it is really
fast and nearly maintenance free and has no further dependencies (so you
don't need to be a "big data engineer" to benefit from it).

This can be very useful to investigate customer complaints of DNS slowness,
or that a domain was down etc. It is also extremely powerful for finding
infected users. A commandline like:

$ dgrep t=week pr=spamhaus-dbl | jq ".items[].origRequestor"  | sort | uniq -c \
| sort -rn | head -10

.. will find in a few seconds the top-10 IP addresses that over the past week
had the most queries  blocked by the 'spamhaus-dbl' RPZ. The output of dgrep is JSON,
easily queried and selected by jq.

But again - I don't want to promote our commercial Platform offering here
too much.  For the open source world, you should be able to bake up a
solution based on elastic search, kibana etc that ingests our protobufs.


        Bert




--
Ciro Iriarte
http://iriarte.it
--

_______________________________________________
Pdns-users mailing list
[hidden email]
https://mailman.powerdns.com/mailman/listinfo/pdns-users
Loading...