mirror of
https://github.com/sogou/workflow.git
synced 2026-10-26 09:39:24 +08:00
Merge branch 'master' of https://github.com/sogou/workflow into nossl
This commit is contained in:
Xie Han
+47
-77
@@ -1,100 +1,70 @@
|
||||
# 关于DNS
|
||||
当使用域名请求网络时，首先需要通过域名解析获取服务器地址，再使用网络地址进行后续的请求。Workflow已经实现了完备的域名解析和缓存系统，通常来说用户无需知晓内部机制即可流畅地发起网络任务。
|
||||
|
||||
DNS(域名服务协议)是一种分布式网络目录服务，主要用于域名与IP地址的相互转换。
|
||||
在进行通信访问的时候，需要对非IP的域名进行DNS解析，这个过程就是域名到IP地址的转换过程。
|
||||
DNS解析是一个比较大的消耗，不管是服务器还是本地操作系统，通常都会有自己的DNS Cache负责减少不必要的请求。
|
||||
有一些程序也会在自己的进程内设计自己的DNS Cache，包括常见流行的浏览器、通信框架等。
|
||||
workflow也设计了自己的DNS Cache，为了方便用户使用，DNS这部分功能被框架完全接管“隐藏”了起来。
|
||||
## DNS相关配置
|
||||
Workflow中的全局配置包括
|
||||
|
||||
### TTL
|
||||
|
||||
全称是“生存时间（Time To Live)”，简单的说TTL表示DNS记录在DNS Cache上缓存的时间。
|
||||
|
||||
### 框架的DNS方法
|
||||
|
||||
#### 同步请求
|
||||
通过调用系统函数getaddrinfo获取结果，一些细节：
|
||||
1. 当命中框架自己的DNS Cache且TTL有效时，DNS解析不会发生。
|
||||
2. 当域名是ipv4、ipv6、unix-domain-socket，DNS解析不会发生。
|
||||
3. DNS解析是一个特殊的计算任务,被封装成了一个WFThreadTask。
|
||||
4. DNS解析使用的是一个完全独立隔离的线程池，即不占用计算线程池、也不占用通信线程池。
|
||||
|
||||
#### 异步请求
|
||||
框架实现了完备的DNS协议解析，在Unix系统下默认将默认使用框架内置的DNS解析器，不会创建DNS线程池。
|
||||
如需想恢复为多线程dns解析，需要在全局配置里将resolv_conf_path参数设置为NULL。
|
||||
|
||||
### 全局DNS配置
|
||||
|
||||
在[WFGlobal.h](../src/manager/WFGlobal.h)文件里，可以看到我们一个全局配置信息：
|
||||
~~~cpp
|
||||
struct WFGlobalSettings
|
||||
{
|
||||
EndpointParams endpoint_params;
|
||||
struct EndpointParams endpoint_params;
|
||||
struct EndpointParams dns_server_params;
|
||||
unsigned int dns_ttl_default;
|
||||
unsigned int dns_ttl_min;
|
||||
int dns_threads;
|
||||
int poller_threads;
|
||||
int handler_threads;
|
||||
int compute_threads;
|
||||
int fio_max_events;
|
||||
const char *resolv_conf_path;
|
||||
const char *hosts_path;
|
||||
};
|
||||
|
||||
static constexpr struct WFGlobalSettings GLOBAL_SETTING_DEFAULT =
|
||||
{
|
||||
.endpoint_params = ENDPOINT_PARAMS_DEFAULT,
|
||||
.dns_server_params = ENDPOINT_PARAMS_DEFAULT,
|
||||
.dns_ttl_default = 12 * 3600, /* in seconds */
|
||||
.dns_ttl_min = 180, /* reacquire when communication error */
|
||||
.dns_threads = 4,
|
||||
.poller_threads = 4,
|
||||
.handler_threads = 20,
|
||||
.compute_threads = -1,
|
||||
.resolv_conf_path = "/etc/resolv.conf",
|
||||
.hosts_path = "/etc/hosts",
|
||||
};
|
||||
~~~
|
||||
其中，与DNS相关的配置包括：
|
||||
* dns_server_params：对dns服务器的最大并发数，超时等配置。
|
||||
* dns_threads: DNS线程池线程数，默认4。只有当resolv_conf_path配置为空时，这个参数才会起作用。否则我们并不会创建dns线程。
|
||||
* dns_ttl_default: DNS Cache中默认的TTL，单位秒，默认12小时，dns cache是当前进程的，即进程退出就会消失，配置也仅对当前进程有效。
|
||||
* dns_ttl_min: dns最短生效时间，单位秒，默认3分钟，用于通信失败重试是否尝试重新dns的决策。
|
||||
* resolv_conf_path: resolv.conf配置文件路径，为NULL表示使用多线程DNS解析。
|
||||
* hosts_path: hosts配置文件路径。可以为NULL。
|
||||
|
||||
简单来讲，每次通信都会检查TTL来决定要不要重新进行DNS解析。
|
||||
默认检查dns_ttl_default，通信失败重试时才会去检查dns_ttl_min。
|
||||
其中与域名解析相关的配置项有
|
||||
|
||||
全局的DNS配置，可以通过upstream功能，被单独的地址配置覆盖。
|
||||
Upstream每一个AddressParams也有dns_ttl_default和dns_ttl_min配置项，使用方式与Global相仿。
|
||||
具体结构详见[upstream文档](./about-upstream.md#Address属性)。
|
||||
* dns_server_params
|
||||
* address_family: 该项会在后续展开说明
|
||||
* max_connections: 向DNS服务器发送请求的最大并发数，默认为200
|
||||
* connect_timeout/response_timeout/ssl_connect_timeout: 参考[超时](about-timeout.md)相关说明
|
||||
* dns_threads: 当使用同步方式实现域名解析时，解析操作会在独立的线程池中执行，该项指定线程池的线程数，默认为4
|
||||
* dns_ttl_default: 域名解析成功的结果会被放到域名缓存中，该项指定其存活时间，单位为秒，默认值12小时，当解析结果过期后会重新解析以获取最新内容
|
||||
* dns_ttl_min: 当通信失败时，有可能出现缓存的结果已经失效的情况，该项指定一个较短的存活时间，当通信失败时以更频繁的速率更新缓存，单位为秒，默认值3分钟
|
||||
* resolv_conf_path: 该文件保存了访问DNS相关的配置，在常见的Linux发行版上通常位于`/etc/resolv.conf`，若该项配置为`NULL`则表示使用多线程同步解析的模式
|
||||
* hosts_path: 该文件是一个本地的域名查找表，若被解析的域名命中该表则不会向DNS发起请求，在常见的Linux发行版上通常位于`/etc/hosts`，若该项配置为`NULL`则表示不使用查找表
|
||||
|
||||
### resolv.conf扩展功能
|
||||
Workflow对`resolv.conf`配置文件进行了扩展，用户可以通过修改配置以支持`DNS over TLS(DoT)`功能，**注意**直接修改`/etc/resolv.conf`会影响其他进程，可以将该文件复制一份用于修改，并将Workflow的`resolv_conf_path`配置修改为新文件的路径。例如使用`dnss`协议的`nameserver`会通过SSL进行连接
|
||||
|
||||
## SSL DNS (DoT)的支持
|
||||
我们的主分支代码（不包括windows分支）支持通过SSL连接访问DNS服务器，即DoT。
|
||||
我们简单的扩展了**resolv.conf**的格式，你可以通过以下方式加入一个SSL dns server:
|
||||
~~~bash
|
||||
nameserver dnss://8.8.8.8/
|
||||
~~~
|
||||
我们用**dnss://** 来表示一个SSL dns server地址。通过以上的配置，我们所有的DNS请求都将通过SSL连接与全球DNS server 8.8.8.8通信。
|
||||
全球DNS server完美支持SSL连接，大家可以立刻试用一下这个功能。
|
||||
为了不用修改系统的**resolv.conf**文件，你可能需要创建一个私有的**resolv.conf**，并修改workflow配置：
|
||||
~~~cpp
|
||||
#include <workflow/WFGlobal.h>
|
||||
#include <workflow/WFTaskFactory.h>
|
||||
|
||||
int main()
|
||||
{
|
||||
struct WFGlobalSettings settings = GLOBAL_SETTINGS_DEFAULT;
|
||||
settings.resolv_conf_path = "./myresolv.conf";
|
||||
WORKFLOW_library_init(&settings);
|
||||
|
||||
WFHttpTask *task = WFTaskFactory::create_http_task("https://www.example.com", ...);
|
||||
...
|
||||
}
|
||||
nameserver dnss://[2001:4860:4860::8888]/
|
||||
~~~
|
||||
|
||||
### DNS解析策略与DNS cache过期策略
|
||||
* 同一个域名，同时只会发送一个DNS请求，这是通过一种异步锁机制实现的。
|
||||
* 只有一种例外，两个不同的upstream指向同一个域名，但分别要求只用IPv4和只用IPv6，可能会同时发起两个请求。
|
||||
* 对DNS server的最大并发，受dns_server_params.max_connections控制，默认为200。
|
||||
* 某个域名DNS cache过期一瞬间，如果多个请求同时需要请求该域名，会有一个任务重新发起DNS请求，并把原cache有效期临时延长5秒，尽量减少DNS cache过期引起的访问中断。
|
||||
### Address Family
|
||||
在某些网络环境下，虽然本机支持IPv6，但因未被分配公网IPv6地址而无法与外部通信（例如本地IPv6地址以`fe80`开始）。此时可以将`endpoint_params.address_family`设置为`AF_INET`来强制域名解析时仅解析IPv4地址。同样的，`resolv.conf`文件中可能同时指定了`nameserver`的IPv4地址和IPv6地址，此时可以将`dns_server_params.address_family`设置为`AF_INET`或`AF_INET6`来强制仅使用IPv4或IPv6地址来访问DNS。
|
||||
|
||||
### 使用Upstream配置
|
||||
全局配置默认对每个域名生效，若需要对某些域名单独指定不同的配置，则可使用[Upstream](./about-upstream.md#Address属性)功能。使用Upstream可以单独指定`dns_ttl_default`、`dns_ttl_min`配置项，以及通过`endpoint_params.address_family`单独指定该域名使用的IP地址类别。
|
||||
|
||||
|
||||
## 域名解析与缓存策略
|
||||
网络任务通常需要通过域名解析获取到需要访问的IP地址，Workflow中域名解析相关策略如下
|
||||
|
||||
1. 检查域名缓存是否有该域名对应的IP地址，若有缓存且未过期，则使用该组IP地址
|
||||
2. 检查域名是否为IPv4、IPv6地址或`Unix Domain Socket`，若是则直接使用该地址，无需发起域名解析
|
||||
3. 检查`hosts_path`文件中是否包含该域名对应的IP地址，若有则直接使用该地址
|
||||
4. 获取异步锁，保证同一域名的解析请求在同一时刻仅发起一次，并向DNS发起解析请求
|
||||
5. 解析成功后会将解析结果保存到当前进程的域名缓存中，以供下次使用，并释放异步锁
|
||||
6. 解析失败后会释放异步锁且将失败原因通知给等在同一个异步锁上的所有任务，通知结束后再发起的新的任务则会再次请求DNS
|
||||
|
||||
许多需要大量发起网络请求的场景都会配备域名缓存组件，如果每次发起网络任务时都向DNS发起解析请求，则DNS必然会不堪重负。Workflow设置了缓存存活时长（dns_ttl_default和dns_ttl_min）来保证缓存会在合理的时间后过期，以及时更新域名的解析结果。当某个域名的缓存项过期后，首先发现过期的任务会将其存活时间延长5秒并向DNS发起解析请求，5秒内同一域名上的请求会直接使用缓存的DNS解析结果，而无需等待本次解析结束。
|
||||
|
||||
异步锁机制可以保证**同一域名**的解析请求在同一时刻仅发起一次，在没有锁保护的情况下，若短时间内对同一域名发起大量网络任务，每个任务都会因无法从缓存中获取结果而向DNS发起解析请求，这会对DNS带来很大且不必要的负担。这里的同一域名表示的是`(host, port, family)`三元组，若通过Upstream的方式对某域名分别要求只使用IPv4和IPv6，则他们会被不同的异步锁保护，也就有可能同时发起DNS请求。
|
||||
|
||||
|
||||
### 异步域名解析
|
||||
Workflow实现了完备的DNS任务（参考[dns_cli](./tutorial-17-dns_cli.md)），若指定了`resolv_conf_path`配置项，则向DNS发起域名解析时会使用异步请求的方式进行，在类Unix系统下，Workflow默认使用`/etc/resolv.conf`作为该配置的值。异步域名解析不会阻塞任何线程，也不会独占线程池，可以更高效地完成域名解析的任务。
|
||||
|
||||
### 同步域名解析
|
||||
若指定`resolv_conf_path`为`NULL`，则会通过调用`getaddrinfo`函数来实现同步域名解析，该方式会使用独立的线程池，其线程数通过`dns_threads`参数配置。若短时间内需要发起较多的域名解析请求，则同步的方式会带来较大的延迟。
|
||||
|
||||
+41
-83
@@ -1,112 +1,70 @@
|
||||
# About DNS
|
||||
When using a domain name to request the network, you first need to obtain the server address through domain name resolution, and then use the network address to make subsequent requests. Workflow has implemented a complete domain name resolution and caching system. Generally speaking, users can initiate network tasks smoothly without knowing the internal mechanism.
|
||||
|
||||
Domain Network System (DNS) is a distributed network directory service. It is mainly used for the conversion between a domain name and an IP address.
|
||||
During the communication access, it is required to perform DNS resolution for a non-IP domain name. DNS resolution is the process that converts a domain name to an IP address.
|
||||
DNS resolution is resource consuming. The operating systems on both servers and local computers usually have their own DNS Cache to reduce unnecessary requests.
|
||||
Some programs, including popular browsers and communication frameworks, also have their own DNS Cache in their processes. Workflow is also designed with its own DNS Cache. For convenience purposes, DNS features are completely transparent and “hidden” in the framework.
|
||||
|
||||
### TTL
|
||||
|
||||
TTL (Time To Live) refers to the amount of time a DNS record is considered up-to-date in the DNS Cache.
|
||||
|
||||
### DNS methods in the framework
|
||||
|
||||
#### Sync request method
|
||||
Currently the framework directly calls the system function **getaddrinfo** to resolve domain names. Some details:
|
||||
|
||||
1. When the DNS Cache of the framework is hit and its TTL is valid, DNS resolution will not happen.
|
||||
2. When the domain name belongs to IPv4, IPv6 or unix-domain-socket, DNS resolution will not happen.
|
||||
3. DNS resolution is a special computing task, which is encapsulated as a WFThreadTask.
|
||||
4. DNS resolution uses a completely independent and isolated thread pool, that is, it does not occupy the computing thread pool or the communication thread pool.
|
||||
|
||||
#### Async request method
|
||||
The framework implements a complete DNS protocol resolution. The current version requires the user to manually specify `resolv_conf_path` to enable this method. On common Linux distributions, `resolv_conf_path` is generally `/etc/resolv.conf`, and `hosts_path` is generally `/etc/hosts`. Leave the above parameters blank, and the synchronous request method will be used.
|
||||
|
||||
### Global DNS configuration
|
||||
|
||||
You can see global settings in [WFGlobal.h](/src/manager/WFGlobal.h).
|
||||
## DNS related configuration
|
||||
Global configuration in Workflow includes
|
||||
|
||||
~~~cpp
|
||||
struct WFGlobalSettings
|
||||
{
|
||||
EndpointParams endpoint_params;
|
||||
struct EndpointParams endpoint_params;
|
||||
struct EndpointParams dns_server_params;
|
||||
unsigned int dns_ttl_default;
|
||||
unsigned int dns_ttl_min;
|
||||
int dns_threads;
|
||||
int poller_threads;
|
||||
int handler_threads;
|
||||
int compute_threads;
|
||||
int fio_max_events;
|
||||
const char *resolv_conf_path;
|
||||
const char *hosts_path;
|
||||
};
|
||||
|
||||
static constexpr struct WFGlobalSettings GLOBAL_SETTING_DEFAULT =
|
||||
{
|
||||
.endpoint_params = ENDPOINT_PARAMS_DEFAULT,
|
||||
.dns_ttl_default = 12 * 3600, /* in seconds */
|
||||
.dns_ttl_min = 180, /* reacquire when communication error */
|
||||
.dns_threads = 4,
|
||||
.poller_threads = 4,
|
||||
.handler_threads = 20,
|
||||
.compute_threads = -1,
|
||||
.resolv_conf_path = "/etc/resolv.conf",
|
||||
.hosts_path = "/etc/hosts",
|
||||
};
|
||||
~~~
|
||||
|
||||
where the DNS-related configuration items include:
|
||||
Among them, the configuration items related to domain name resolution include
|
||||
|
||||
* dns\_threads: the number of threads in the DNS thread pool, 4 by default. This item will be ignored if the resolv_conf_path is not NULL, and no DNS thread will be created.
|
||||
* dns\_ttl\_default: default TTL in DNS Cache in seconds, 12 hours by default; DNS cache is used by the current process, and will be destroyed when the process exists. The configuration is valid only for the current process.
|
||||
* dns\_ttl\_min: minimum DNS ttl value, in seconds, 3 minutes by default, which is used to decide whether to retry DNS resolution after communication failure.
|
||||
* resolv_conf_path: Path of the resolv.conf configuration file, we will use multi-threaded DNS resolution when it is NULL.
|
||||
* hosts_path: Path of the hosts configuration file. This item can also be NULL.
|
||||
* dns_server_params
|
||||
* address_family: This item will be explained later
|
||||
* max_connections: The maximum number of concurrent requests sent to the DNS server, the default is 200
|
||||
* connect_timeout/response_timeout/ssl_connect_timeout: refer to [timeout](about-timeout.md) for related instructions
|
||||
* dns_threads: When using synchronous mode to implement domain name resolution, the resolution operation will be executed in an independent thread pool. This item specifies the number of threads in the thread pool. The default is 4.
|
||||
* dns_ttl_default: The result of successful domain name resolution will be placed in the domain name cache. This item specifies its survival time in seconds. The default value is 12 hours. When the resolution result expires, it will be re-parsed to obtain the latest content.
|
||||
* dns_ttl_min: When communication fails, the cached result may have expired. This item specifies a shorter survival time. When communication fails, the cache is updated at a more frequent rate. The unit is seconds. The default value is 3 minutes.
|
||||
* resolv_conf_path: This file saves the configuration related to accessing DNS. It is usually located in `/etc/resolv.conf` on common Linux distributions. If this item is configured as `NULL`, it means using multi-threaded synchronous resolution mode.
|
||||
* hosts_path: This file is a local domain name lookup table. If the resolved domain name hits this table, it will not initiate a request to DNS. It is usually located in `/etc/hosts` on common Linux distributions. If this item is configured as `NULL` means not to use the lookup table
|
||||
|
||||
To put it simply, in every communication, the system will check TTL to decide whether to refresh DNS resolution.
|
||||
dns\_ttl\_default is checked by default, and dns\_ttl\_min is checked in the retry after communication failure.
|
||||
### resolv.conf extensions
|
||||
Workflow has extended the `resolv.conf` configuration file. Users can modify the configuration to support the `DNS over TLS(DoT)`. **Note** directly modifying `/etc/resolv.conf` will affect other processes. You can make a copy of the file for modification, and modify the `resolv_conf_path` configuration of Workflow to the path of the new file. For example, a `nameserver` using the `dnss` protocol will connect via SSL
|
||||
|
||||
The global DNS configuration can be overridden by the configuration for an individual address in the upstream.
|
||||
In Upstream, each AddressParams can also have its own dns\_ttl\_default and dns\_ttl\_min, and you can configure them in the same way as you configure the Global items.
|
||||
For the detailed structures, please see [upstream documents](/docs/en/about-upstream.md#Address).
|
||||
|
||||
### DNS over SSL (Dot)
|
||||
|
||||
On the master branch (not available on windows branch), we also supports DNS resolving over SSL (Dot) by extending the format of **resolv.conf** slightly. You may edit the **resolv.conf** file and add an SSL dns server like:
|
||||
~~~bash
|
||||
nameserver dnss://8.8.8.8/
|
||||
~~~
|
||||
We use the **dnss://** scheme to represent an SSL dns server address. With the config abort, all DNS resolving will use SSL connection to global dns server 8.8.8.8. The global DNS supports SSL perfectly, and you can try this feature right now! But to avoid changing the system **resolv.conf** file (/etc/resolv.conf), you may need to setup a private **resolv.conf** path:
|
||||
~~~cpp
|
||||
#include <workflow/WFGlobal.h>
|
||||
#include <workflow/WFTaskFactory.h>
|
||||
|
||||
int main()
|
||||
{
|
||||
struct WFGlobalSettings settings = GLOBAL_SETTINGS_DEFAULT;
|
||||
settings.resolv_conf_path = "./myresolv.conf";
|
||||
WORKFLOW_library_init(&settings);
|
||||
|
||||
WFHttpTask *task = WFTaskFactory::create_http_task("https://www.example.com", ...);
|
||||
...
|
||||
}
|
||||
nameserver dnss://[2001:4860:4860::8888]/
|
||||
~~~
|
||||
|
||||
### Handling at TTL expiration moment under high concurrency
|
||||
### Address Family
|
||||
In some network environments, although the machine supports IPv6, it cannot communicate with the outside because it has not been assigned a public IPv6 address (for example, the local IPv6 address starts with `fe80`). At this time, you can set `endpoint_params.address_family` to `AF_INET` to force only IPv4 addresses to be resolved during domain name resolution. Similarly, the `resolv.conf` file may specify both the IPv4 address and the IPv6 address of the `nameserver`. In this case, you can set `dns_server_params.address_family` to `AF_INET` or `AF_INET6` to force the use of only IPv4 or IPv6 addresses to access DNS.
|
||||
|
||||
At the moment when the TTL is exceeded, if a large number of concurrent requests are sent to a domain name, a large amount of DNS resolution for that domain name may occur at the same time.
|
||||
The framework uses a self-consistent logic to reasonably avoid/reduce this possibility:
|
||||
### Use Upstream configuration
|
||||
The global configuration takes effect for each domain name by default. If you need to specify different configurations for certain domain names, you can use the [Upstream](./about-upstream.md#Address attribute) function. Using Upstream, you can individually specify the `dns_ttl_default` and `dns_ttl_min` configuration items, and individually specify the IP address family used by the domain name through `endpoint_params.address_family`.
|
||||
|
||||
* When the results are obtained from DNS Cache, if the TTL is exceeded, the TTL will be increased by 10 seconds, and then the TTL expiration will be returned. All happen under the protection of a Mutex.
|
||||
* If a large number of requests flood in at the moment of TTL expiration, under the protection of this Mutex, \[the first request] will get the expired results and initiate DNS resolution, while other requests will continue to use the old results within 10 seconds.
|
||||
* As long as the new DNS resolution for \[the first request] is successfully completed within 10 seconds, the DNS Cache is updated to ensure the correctness of the logic; in the next 10 seconds, there will be only one DNS resolution.
|
||||
* In every ten seconds, there will be only one DNS resolution for the “recently” expired domain name.
|
||||
* In order to prevent this mutual exclusion logic from affecting performance, the framework uses the double-checked locks to accelerate processing and effectively avoid the competition of Mutex locks.
|
||||
* Once again, please note that it is only valid for the "just" expired DNS records, and has no impact for the DNS records that expires long time ago.
|
||||
* For further information on the logic of this part, please see the source codes of [DNSCache](/src/manager/DNSCache.h).
|
||||
|
||||
Currently, there are still two scenarios in which the framework has to perform a large number of DNS resolutions for the same domain name at the same time:
|
||||
## Domain name resolution and caching strategy
|
||||
Network tasks usually require domain name resolution to obtain the IP address that needs to be accessed. The relevant strategies for domain name resolution in Workflow are as follows:
|
||||
|
||||
1. The program just started, and instantly made a large number of requests to the same domain name.
|
||||
2. A domain name has not been visited for a long time (far larger than TTL), and suddenly a lot of requests are made to this domain name.
|
||||
1. Check whether the domain name cache has the IP address corresponding to the domain name. If there is a cache and it has not expired, use this set of IP addresses.
|
||||
2. Check whether the domain name is an IPv4, IPv6 address or `Unix Domain Socket`. If so, use the address directly without initiating domain name resolution.
|
||||
3. Check whether the `hosts_path` file contains the IP address corresponding to the domain name. If so, use the address directly.
|
||||
4. Obtain an asynchronous lock to ensure that a resolution request for the same domain name is only initiated once at the same time, and initiate a resolution request to DNS
|
||||
5. After successful parsing, the parsing result will be saved to the domain name cache of the current process for next use, and the asynchronous lock will be released.
|
||||
6. After the parsing fails, the asynchronous lock will be released and the failure reason will be notified to all tasks waiting on the same asynchronous lock. New tasks initiated after the notification is completed will request DNS again.
|
||||
|
||||
We think that these two scenarios are acceptable in the framework. To put it more precisely, a large number of DNS requests in this scenario are completely reasonable and logical.
|
||||
Many scenarios that require a large number of network requests will be equipped with a domain name caching component. If a resolution request is sent to the DNS every time a network task is initiated, the DNS will inevitably be overwhelmed. Workflow sets the cache survival time (dns_ttl_default and dns_ttl_min) to ensure that the cache will expire after a reasonable period of time and the domain name resolution results can be updated in a timely manner. When a cache item of a domain name expires, the first task found to be expired will extend its survival time by 5 seconds and initiate a resolution request to DNS. Requests on the same domain name within 5 seconds will directly use the cached DNS resolution results without waiting.
|
||||
|
||||
The asynchronous lock mechanism can ensure that the resolution request for the **same domain name** is only initiated once at the same time. Without lock protection, if a large number of network tasks are initiated for the same domain name in a short period of time, each task will be unable to be retrieved from the cache. Too many resolution request to DNS will place a large and unnecessary burden on DNS. The same domain name here represents the `(host, port, family)` triplet. If a domain name is required to only use IPv4 and IPv6 through Upstream, they will be protected by different asynchronous locks, and it is possible to request DNS at the same time.
|
||||
|
||||
|
||||
### Asynchronous domain name resolution
|
||||
Workflow implements a complete DNS task. If the `resolv_conf_path` configuration item is specified, an asynchronous request will be used when initiating domain name resolution to DNS. Under Unix-like systems, Workflow uses `/etc/resolv.conf` as the value of this configuration by default. Asynchronous domain name resolution does not block any threads or monopolize the thread pool, and can complete the task of domain name resolution more efficiently.
|
||||
|
||||
### Synchronous domain name resolution
|
||||
If `resolv_conf_path` is specified as `NULL`, synchronous domain name resolution will be achieved by calling the `getaddrinfo` function. This method will use an independent thread pool, and the number of threads is configured through the `dns_threads` parameter. If a large number of domain name resolution requests need to be initiated in a short period of time, the synchronization method will cause a large delay.
|
||||
|
||||
+12
-17
@@ -380,23 +380,18 @@ static double __evaluate_json_number(const char *integer,
|
||||
}
|
||||
}
|
||||
|
||||
num = mant;
|
||||
if (exp != 0 && figures != 0)
|
||||
{
|
||||
if (exp > 309 - figures)
|
||||
num = INFINITY;
|
||||
else if (exp > 0)
|
||||
num *= __power_of_10[exp];
|
||||
else if (exp > -309)
|
||||
num /= __power_of_10[-exp];
|
||||
else if (exp > -324 - figures)
|
||||
{
|
||||
num /= __power_of_10[-exp - 308];
|
||||
num /= __power_of_10[308];
|
||||
}
|
||||
else
|
||||
num = 0.0;
|
||||
}
|
||||
if (exp == 0 || figures == 0)
|
||||
num = mant;
|
||||
else if (exp > 291)
|
||||
num = INFINITY;
|
||||
else if (exp > 0)
|
||||
num = mant * __power_of_10[exp];
|
||||
else if (exp > -309)
|
||||
num = mant / __power_of_10[-exp];
|
||||
else if (exp > -324 - figures)
|
||||
num = mant / __power_of_10[-exp - 308] / __power_of_10[308];
|
||||
else
|
||||
num = 0.0;
|
||||
|
||||
return sign ? -num : num;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user