failed to flush chunk

Fri, Mar 25 2022 3:08:32 pm | [2022/03/25 07:08:32] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Match kube. Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [debug] [out coro] cb_destroy coro_id=18 Fri, Mar 25 2022 3:08:22 pm | [2022/03/25 07:08:22] [debug] [outputes.0] task_id=4 assigned to thread #0 fluent-bit-1.6.10 Log loss failed to flush chunk. Hi @yangtian9999 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"BuMmun8BI6SaBP9l_8rZ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamicall Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Fri, Mar 25 2022 3:08:44 pm | [2022/03/25 07:08:44] [debug] [task] created task=0x7ff2f183afc0 id=17 OK Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=34055641 removing file name /var/log/containers/hello-world-bjfnf_argo_main-0b26876c79c5790bdaf62ba2d9512269459746b1c5711a6445256dc5a4930b65.log [2022/03/24 04:19:20] [debug] [input chunk] tail.0 is paused, cannot append records Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [retry] new retry created for task_id=12 attempts=1 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Tried to remove everything and leave only in_tail and out_es. Fri, Mar 25 2022 3:08:31 pm | [2022/03/25 07:08:31] [debug] [outputes.0] task_id=9 assigned to thread #0 Fri, Mar 25 2022 3:08:36 pm | [2022/03/25 07:08:36] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 is now available [2022/03/24 04:20:00] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Fri, Mar 25 2022 3:08:42 pm | [2022/03/25 07:08:42] [debug] [retry] new retry created for task_id=16 attempts=1 Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY helm install helm-charts-fluent-bit-0.19.19. [2022/03/24 04:20:06] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000 Failed to create target, ioutil.ReadDir: readdirent: not a directory. [2022/03/24 04:20:49] [debug] [upstream] KA connection #102 to 10.3.4.84:9200 has been assigned (recycled) "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"KOMoun8BI6SaBP9lIP7t","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [ warn] [engine] failed to flush chunk '1-1648192103.858183.flb', retry in 30 seconds: task_id=5, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [upstream] KA connection #36 to 10.3.4.84:9200 has been assigned (recycled) Match host. Trace logging is enabled but there is no log entry to help me further. Sign in [2022/03/24 04:21:06] [debug] [outputes.0] task_id=1 assigned to thread #0 [2022/03/24 04:19:54] [debug] [retry] re-using retry for task_id=0 attempts=3 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [debug] [out coro] cb_destroy coro_id=20 Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [out coro] cb_destroy coro_id=6 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. [2022/03/24 04:19:20] [debug] [input chunk] tail.0 is paused, cannot append records "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"ZuMnun8BI6SaBP9lLtm1","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [outputes.0] task_id=8 assigned to thread #0 [2022/03/22 03:57:47] [ warn] [engine] failed to flush chunk '1-1647920812.174022994.flb', retry in 746 seconds: task_id=619, input=tail.0 > output=es.0 (out_id=0) Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. [2022/03/24 04:20:20] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/argo-server-6d7cf9c977-dlwnk_argo_argo-server-7e1ccfbd60b7539a1b2984f2f46de601d567ce83e87d434e173df195e44b5224.log, inode 101715266 Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [ warn] [engine] failed to flush chunk '1-1648192098.623024610.flb', retry in 16 seconds: task_id=1, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:46 pm | [2022/03/25 07:08:46] [debug] [retry] re-using retry for task_id=12 attempts=2 Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [debug] [upstream] KA connection #36 to 10.3.4.84:9200 is now available "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"yuMnun8BI6SaBP9lo-jn","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Improve this answer. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [debug] [input chunk] update output instances with new chunk size diff=1083 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [retry] re-using retry for task_id=8 attempts=2 Fri, Mar 25 2022 3:08:47 pm | [2022/03/25 07:08:47] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 has been assigned (recycled) "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"BOMmun8BI6SaBP9l_8rZ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Hi @yangtian9999. Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=35359369 events: IN_ATTRIB Bug Report Describe the bug Continuing logging in pod fluent-bit-84pj9 [2022/03/22 03:48:51] [ warn] [engine] failed to flush chunk '1-1647920930.175942635.flb', retry in 11 seconds: task_i. Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 is now available [2022/03/24 04:20:20] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/svclb-traefik-twmt7_kube-system_lb-port-80-10ce439b02864f9075c8e41c716e394a6a6cda391ae753798cde988271ff35ef.log, inode 67186751 Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [input chunk] update output instances with new chunk size diff=650 Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [out coro] cb_destroy coro_id=5 Fri, Mar 25 2022 3:08:29 pm | [2022/03/25 07:08:29] [debug] [outputes.0] task_id=7 assigned to thread #0 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"feMnun8BI6SaBP9l3vN-","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [outputes.0] task_id=20 assigned to thread #1 We are trying to get EKS logs to Graylog. Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [task] created task=0x7ff2f183ac00 id=15 OK Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [debug] [retry] new retry created for task_id=18 attempts=1 Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Host 10.3.4.84 Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [debug] [out coro] cb_destroy coro_id=3 Fri, Mar 25 2022 3:08:23 pm | [2022/03/25 07:08:23] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Fri, Mar 25 2022 3:08:36 pm | [2022/03/25 07:08:36] [debug] [http_client] not using http_proxy for header [2022/03/22 03:57:48] [ warn] [engine] failed to flush chunk '1-1647920969.178403746.flb', retry in 130 seconds: task_id=774, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [ warn] [engine] failed to flush chunk '1-1648192113.5409018.flb', retry in 7 seconds: task_id=11, input=tail.0 > output=es.0 (out_id=0) [2022/03/24 04:19:59] [debug] [upstream] KA connection #102 to 10.3.4.84:9200 has been assigned (recycled) If I send the CONT signal to fluentbit I see that fluentbit still has them. Fri, Mar 25 2022 3:08:31 pm | [2022/03/25 07:08:31] [debug] [http_client] not using http_proxy for header "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"0uMnun8BI6SaBP9lo-jn","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"J-Mmun8BI6SaBP9lh4vZ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamicall Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 is now available Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [debug] [input chunk] update output instances with new chunk size diff=650 [2022/03/24 04:19:30] [debug] [retry] re-using retry for task_id=2 attempts=2 Expected behavior logs from the source folder should've been transferred to elasticsearch. Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [input chunk] update output instances with new chunk size diff=657 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:37 pm | [2022/03/25 07:08:37] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 has been assigned (recycled) [2022/03/24 04:19:50] [debug] [upstream] KA connection #102 to 10.3.4.84:9200 is now available [2022/03/24 04:19:38] [debug] [retry] re-using retry for task_id=0 attempts=2 You're sending more data than the cluster can index. Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [http_client] not using http_proxy for header To those having this same issue, can you share your config and log files with debug level enabled? Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [debug] [out coro] cb_destroy coro_id=11 [2022/03/24 04:19:50] [debug] [outputes.0] HTTP Status=200 URI=/_bulk [2022/03/24 04:19:49] [ warn] [engine] failed to flush chunk '1-1648095560.254537600.flb', retry in 15 seconds: task_id=1, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:37 pm | [2022/03/25 07:08:37] [debug] [outputes.0] task_id=7 assigned to thread #1

Stark County Il Police Scanner, Vulcraft Deck Cad Details, Does Jesse Know Walt Killed Jane, Articles F

failed to flush chunk