Cilium Enterprise integration for Grafana Cloud
The Cilium Enterprise integration uses the Grafana Agent to collect metrics exposed by the Cilium Operator, Cilium Agent and its components, as well as Hubble. A series of dashboards have been provided, both for overviews and per-component basis. This integration includes 18 useful alerts and 20 pre-built dashboards to help monitor and visualize Cilium Enterprise metrics.
Before you begin
Please note, this integration is intended exclusively for use with Cilium Enterprise, and does not support the Cilium OSS product.
This integration monitors a Cilium Enterprise & Hubble Enterprise deployment that has metrics exporters enabled. Please ensure you have completed the following setup steps:
- Enabled the embedded Prometheus exporter in your Cilium deployment to collect and expose metrics
- Enabled the embedded Prometheus exporter in Hubble if you want Hubble metrics to be included.
Once the exporters have been enabled the metrics are automatically exposed and available to be scraped by either Prometheus or a Grafana Alloy deployed to the cluster.
The following sample using Helm enables the Prometheus metrics endpoint and configures the relevant metrics for Cilium Agent, Cilium Operator, and Hubble:
helm install cilium cilium/cilium --version 1.12.2 \
--namespace kube-system \
--set prometheus.enabled=true \
--set operator.prometheus.enabled=true
Sample Hubble metrics setup using helm, specifying which metrics to enable capture and export for:
helm install cilium cilium/cilium --version 1.12.2 \
--namespace kube-system \
--set hubble.metrics.enabled="{dns,drop,tcp,flow,icmp,http}"
Install Cilium Enterprise integration for Grafana Cloud
- In your Grafana Cloud stack, click Connections in the left-hand menu.
- Find Cilium Enterprise and click its tile to open the integration.
- Review the prerequisites in the Configuration Details tab and set up Grafana Agent to send Cilium Enterprise metrics to your Grafana Cloud instance.
- Click Install to add this integration’s pre-built dashboards and alerts to your Grafana Cloud instance, and you can start monitoring your Cilium Enterprise setup.
Configuration snippets for Grafana Alloy
Simple mode
These snippets are configured to scrape a single Cilium Enterprise instance running locally with default ports.
First, manually copy and append the following snippets into your alloy configuration file.
Metrics snippets
discovery.kubernetes "metrics_integrations_integrations_cilium_enterprise_hubble_timescape_ingestor" {
role = "endpoints"
}
discovery.kubernetes "metrics_integrations_integrations_cilium_enterprise_hubble_timescape_server" {
role = "endpoints"
}
discovery.kubernetes "metrics_integrations_integrations_cilium_enterprise_cilium_agent" {
role = "endpoints"
}
discovery.kubernetes "metrics_integrations_integrations_cilium_enterprise_cilium_operator" {
role = "endpoints"
}
discovery.kubernetes "metrics_integrations_integrations_cilium_enterprise_hubble" {
role = "endpoints"
}
discovery.kubernetes "metrics_integrations_integrations_cilium_enterprise_hubble_enterprise" {
role = "endpoints"
}
discovery.relabel "metrics_integrations_integrations_cilium_enterprise_hubble_timescape_ingestor" {
targets = discovery.kubernetes.metrics_integrations_integrations_cilium_enterprise_hubble_timescape_ingestor.targets
rule {
source_labels = ["__meta_kubernetes_service_label_app_kubernetes_io_instance", "__meta_kubernetes_service_labelpresent_app_kubernetes_io_instance"]
regex = "(hubble-timescape);true"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_service_label_app_kubernetes_io_name", "__meta_kubernetes_service_labelpresent_app_kubernetes_io_name"]
regex = "(hubble-timescape-ingestor);true"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_port_name"]
regex = "metrics"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_address_target_kind", "__meta_kubernetes_endpoint_address_target_name"]
regex = "Node;(.*)"
target_label = "node"
replacement = "${1}"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_address_target_kind", "__meta_kubernetes_endpoint_address_target_name"]
regex = "Pod;(.*)"
target_label = "pod"
replacement = "${1}"
}
rule {
source_labels = ["__meta_kubernetes_namespace"]
target_label = "namespace"
}
rule {
source_labels = ["__meta_kubernetes_service_name"]
target_label = "service"
}
rule {
source_labels = ["__meta_kubernetes_pod_name"]
target_label = "pod"
}
rule {
source_labels = ["__meta_kubernetes_pod_container_name"]
target_label = "container"
}
rule {
target_label = "endpoint"
replacement = "metrics"
}
rule {
source_labels = ["__address__"]
modulus = 1
target_label = "__tmp_hash"
action = "hashmod"
}
rule {
source_labels = ["__tmp_hash"]
regex = "0"
action = "keep"
}
}
discovery.relabel "metrics_integrations_integrations_cilium_enterprise_hubble_timescape_server" {
targets = discovery.kubernetes.metrics_integrations_integrations_cilium_enterprise_hubble_timescape_server.targets
rule {
source_labels = ["__meta_kubernetes_service_label_app_kubernetes_io_instance", "__meta_kubernetes_service_labelpresent_app_kubernetes_io_instance"]
regex = "(hubble-timescape);true"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_service_label_app_kubernetes_io_name", "__meta_kubernetes_service_labelpresent_app_kubernetes_io_name"]
regex = "(hubble-timescape-server);true"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_port_name"]
regex = "metrics"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_address_target_kind", "__meta_kubernetes_endpoint_address_target_name"]
regex = "Node;(.*)"
target_label = "node"
replacement = "${1}"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_address_target_kind", "__meta_kubernetes_endpoint_address_target_name"]
regex = "Pod;(.*)"
target_label = "pod"
replacement = "${1}"
}
rule {
source_labels = ["__meta_kubernetes_namespace"]
target_label = "namespace"
}
rule {
source_labels = ["__meta_kubernetes_service_name"]
target_label = "service"
}
rule {
source_labels = ["__meta_kubernetes_pod_name"]
target_label = "pod"
}
rule {
source_labels = ["__meta_kubernetes_pod_container_name"]
target_label = "container"
}
rule {
target_label = "endpoint"
replacement = "metrics"
}
rule {
source_labels = ["__address__"]
modulus = 1
target_label = "__tmp_hash"
action = "hashmod"
}
rule {
source_labels = ["__tmp_hash"]
regex = "0"
action = "keep"
}
}
discovery.relabel "metrics_integrations_integrations_cilium_enterprise_cilium_agent" {
targets = discovery.kubernetes.metrics_integrations_integrations_cilium_enterprise_cilium_agent.targets
rule {
source_labels = ["__meta_kubernetes_service_label_k8s_app", "__meta_kubernetes_service_labelpresent_k8s_app"]
regex = "(cilium);true"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_port_name"]
regex = "metrics"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_address_target_kind", "__meta_kubernetes_endpoint_address_target_name"]
regex = "Node;(.*)"
target_label = "node"
replacement = "${1}"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_address_target_kind", "__meta_kubernetes_endpoint_address_target_name"]
regex = "Pod;(.*)"
target_label = "pod"
replacement = "${1}"
}
rule {
source_labels = ["__meta_kubernetes_namespace"]
target_label = "namespace"
}
rule {
source_labels = ["__meta_kubernetes_service_name"]
target_label = "service"
}
rule {
source_labels = ["__meta_kubernetes_pod_name"]
target_label = "pod"
}
rule {
source_labels = ["__meta_kubernetes_pod_container_name"]
target_label = "container"
}
rule {
source_labels = ["__meta_kubernetes_service_label_k8s_app"]
regex = "(.+)"
target_label = "k8s_app"
replacement = "${1}"
}
rule {
target_label = "endpoint"
replacement = "metrics"
}
rule {
source_labels = ["__address__"]
modulus = 1
target_label = "__tmp_hash"
action = "hashmod"
}
rule {
source_labels = ["__tmp_hash"]
regex = "0"
action = "keep"
}
}
discovery.relabel "metrics_integrations_integrations_cilium_enterprise_cilium_operator" {
targets = discovery.kubernetes.metrics_integrations_integrations_cilium_enterprise_cilium_operator.targets
rule {
source_labels = ["__meta_kubernetes_service_label_io_cilium_app", "__meta_kubernetes_service_labelpresent_io_cilium_app"]
regex = "(operator);true"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_service_label_name", "__meta_kubernetes_service_labelpresent_name"]
regex = "(cilium-operator);true"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_port_name"]
regex = "metrics"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_address_target_kind", "__meta_kubernetes_endpoint_address_target_name"]
regex = "Node;(.*)"
target_label = "node"
replacement = "${1}"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_address_target_kind", "__meta_kubernetes_endpoint_address_target_name"]
regex = "Pod;(.*)"
target_label = "pod"
replacement = "${1}"
}
rule {
source_labels = ["__meta_kubernetes_namespace"]
target_label = "namespace"
}
rule {
source_labels = ["__meta_kubernetes_service_name"]
target_label = "service"
}
rule {
source_labels = ["__meta_kubernetes_pod_name"]
target_label = "pod"
}
rule {
source_labels = ["__meta_kubernetes_pod_container_name"]
target_label = "container"
}
rule {
source_labels = ["__meta_kubernetes_service_label_io_cilium_app"]
regex = "(.+)"
target_label = "io_cilium_app"
replacement = "${1}"
}
rule {
target_label = "endpoint"
replacement = "metrics"
}
rule {
source_labels = ["__address__"]
modulus = 1
target_label = "__tmp_hash"
action = "hashmod"
}
rule {
source_labels = ["__tmp_hash"]
regex = "0"
action = "keep"
}
}
discovery.relabel "metrics_integrations_integrations_cilium_enterprise_hubble" {
targets = discovery.kubernetes.metrics_integrations_integrations_cilium_enterprise_hubble.targets
rule {
source_labels = ["__meta_kubernetes_service_label_k8s_app", "__meta_kubernetes_service_labelpresent_k8s_app"]
regex = "(hubble);true"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_port_name"]
regex = "hubble-metrics"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_address_target_kind", "__meta_kubernetes_endpoint_address_target_name"]
regex = "Node;(.*)"
target_label = "node"
replacement = "${1}"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_address_target_kind", "__meta_kubernetes_endpoint_address_target_name"]
regex = "Pod;(.*)"
target_label = "pod"
replacement = "${1}"
}
rule {
source_labels = ["__meta_kubernetes_namespace"]
target_label = "namespace"
}
rule {
source_labels = ["__meta_kubernetes_service_name"]
target_label = "service"
}
rule {
source_labels = ["__meta_kubernetes_pod_name"]
target_label = "pod"
}
rule {
source_labels = ["__meta_kubernetes_pod_container_name"]
target_label = "container"
}
rule {
target_label = "endpoint"
replacement = "hubble-metrics"
}
rule {
source_labels = ["__address__"]
modulus = 1
target_label = "__tmp_hash"
action = "hashmod"
}
rule {
source_labels = ["__tmp_hash"]
regex = "0"
action = "keep"
}
}
discovery.relabel "metrics_integrations_integrations_cilium_enterprise_hubble_enterprise" {
targets = discovery.kubernetes.metrics_integrations_integrations_cilium_enterprise_hubble_enterprise.targets
rule {
source_labels = ["__meta_kubernetes_service_label_app_kubernetes_io_instance", "__meta_kubernetes_service_labelpresent_app_kubernetes_io_instance"]
regex = "(hubble-enterprise);true"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_service_label_app_kubernetes_io_managed_by", "__meta_kubernetes_service_labelpresent_app_kubernetes_io_managed_by"]
regex = "(Helm);true"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_service_label_app_kubernetes_io_name", "__meta_kubernetes_service_labelpresent_app_kubernetes_io_name"]
regex = "(hubble-enterprise);true"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_service_label_helm_sh_chart", "__meta_kubernetes_service_labelpresent_helm_sh_chart"]
regex = "(hubble-enterprise-9999.9999.9999-dev);true"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_port_name"]
regex = "metrics"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_address_target_kind", "__meta_kubernetes_endpoint_address_target_name"]
regex = "Node;(.*)"
target_label = "node"
replacement = "${1}"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_address_target_kind", "__meta_kubernetes_endpoint_address_target_name"]
regex = "Pod;(.*)"
target_label = "pod"
replacement = "${1}"
}
rule {
source_labels = ["__meta_kubernetes_namespace"]
target_label = "namespace"
}
rule {
source_labels = ["__meta_kubernetes_service_name"]
target_label = "service"
}
rule {
source_labels = ["__meta_kubernetes_pod_name"]
target_label = "pod"
}
rule {
source_labels = ["__meta_kubernetes_pod_container_name"]
target_label = "container"
}
rule {
target_label = "endpoint"
replacement = "metrics"
}
rule {
source_labels = ["__meta_kubernetes_pod_node_name"]
target_label = "node"
replacement = "${1}"
}
rule {
source_labels = ["__address__"]
modulus = 1
target_label = "__tmp_hash"
action = "hashmod"
}
rule {
source_labels = ["__tmp_hash"]
regex = "0"
action = "keep"
}
}
prometheus.scrape "metrics_integrations_integrations_cilium_enterprise_hubble_timescape_ingestor" {
targets = discovery.relabel.metrics_integrations_integrations_cilium_enterprise_hubble_timescape_ingestor.output
forward_to = [prometheus.relabel.metrics_integrations_integrations_cilium_enterprise_hubble_timescape_ingestor.receiver]
job_name = "integrations/cilium-enterprise/hubble-timescape-ingestor"
honor_labels = true
scrape_timeout = "1m0s"
}
prometheus.scrape "metrics_integrations_integrations_cilium_enterprise_hubble_timescape_server" {
targets = discovery.relabel.metrics_integrations_integrations_cilium_enterprise_hubble_timescape_server.output
forward_to = [prometheus.relabel.metrics_integrations_integrations_cilium_enterprise_hubble_timescape_server.receiver]
job_name = "integrations/cilium-enterprise/hubble-timescape-server"
honor_labels = true
scrape_timeout = "1m0s"
}
prometheus.scrape "metrics_integrations_integrations_cilium_enterprise_cilium_agent" {
targets = discovery.relabel.metrics_integrations_integrations_cilium_enterprise_cilium_agent.output
forward_to = [prometheus.relabel.metrics_integrations_integrations_cilium_enterprise_cilium_agent.receiver]
job_name = "integrations/cilium-enterprise/cilium-agent"
honor_labels = true
scrape_timeout = "1m0s"
}
prometheus.scrape "metrics_integrations_integrations_cilium_enterprise_cilium_operator" {
targets = discovery.relabel.metrics_integrations_integrations_cilium_enterprise_cilium_operator.output
forward_to = [prometheus.relabel.metrics_integrations_integrations_cilium_enterprise_cilium_operator.receiver]
job_name = "integrations/cilium-enterprise/cilium-operator"
honor_labels = true
scrape_timeout = "1m0s"
}
prometheus.scrape "metrics_integrations_integrations_cilium_enterprise_hubble" {
targets = discovery.relabel.metrics_integrations_integrations_cilium_enterprise_hubble.output
forward_to = [prometheus.relabel.metrics_integrations_integrations_cilium_enterprise_hubble.receiver]
job_name = "integrations/cilium-enterprise/hubble"
honor_labels = true
scrape_timeout = "1m0s"
}
prometheus.scrape "metrics_integrations_integrations_cilium_enterprise_hubble_enterprise" {
targets = discovery.relabel.metrics_integrations_integrations_cilium_enterprise_hubble_enterprise.output
forward_to = [prometheus.relabel.metrics_integrations_integrations_cilium_enterprise_hubble_enterprise.receiver]
job_name = "integrations/cilium-enterprise/hubble-enterprise"
honor_labels = true
scrape_timeout = "1m0s"
}
prometheus.relabel "metrics_integrations_integrations_cilium_enterprise_hubble_timescape_ingestor" {
forward_to = [prometheus.remote_write.metrics_service.receiver]
rule {
source_labels = ["__name__"]
regex = "cilium_agent_api_process_time_seconds_count|cilium_agent_api_process_time_seconds_sum|cilium_api_limiter_processed_requests_total|cilium_bpf_map_ops_total|cilium_bpf_map_pressure|cilium_controllers_runs_duration_seconds_count|cilium_controllers_runs_duration_seconds_sum|cilium_controllers_runs_total|cilium_datapath_conntrack_gc_duration_seconds_count|cilium_datapath_conntrack_gc_duration_seconds_sum|cilium_datapath_conntrack_gc_entries|cilium_datapath_conntrack_gc_key_fallbacks_total|cilium_datapath_conntrack_gc_runs_total|cilium_drop_bytes_total|cilium_drop_count_total|cilium_endpoint_regeneration_time_stats_seconds_count|cilium_endpoint_regeneration_time_stats_seconds_sum|cilium_endpoint_regenerations_total|cilium_endpoint_state|cilium_errors_warnings_total|cilium_forward_bytes_total|cilium_forward_count_total|cilium_identity|cilium_ip_addresses|cilium_k8s_client_api_calls_total|cilium_k8s_client_api_latency_time_seconds_count|cilium_k8s_client_api_latency_time_seconds_sum|cilium_kubernetes_events_received_total|cilium_kubernetes_events_total|cilium_nodes_all_events_received_total|cilium_nodes_all_num|cilium_operator_ces_queueing_delay_seconds_bucket|cilium_operator_ces_sync_errors_total|cilium_operator_ec2_api_duration_seconds_bucket|cilium_operator_identity_gc_entries|cilium_operator_identity_gc_runs|cilium_operator_ipam_allocation_ops|cilium_operator_ipam_deficit_resolver_duration_seconds_bucket|cilium_operator_ipam_interface_creation_ops|cilium_operator_ipam_ips|cilium_operator_ipam_k8s_sync_queued_total|cilium_operator_ipam_nodes|cilium_operator_ipam_resync_queued_total|cilium_operator_ipam_resync_total|cilium_operator_number_of_ceps_per_ces_sum|cilium_operator_process_cpu_seconds_total|cilium_operator_process_open_fds|cilium_operator_process_resident_memory_bytes|cilium_operator_process_virtual_memory_bytes|cilium_policy|cilium_policy_endpoint_enforcement_status|cilium_policy_l7_denied_total|cilium_policy_l7_forwarded_total|cilium_policy_l7_received_total|cilium_proxy_redirects|cilium_proxy_upstream_reply_seconds_count|cilium_proxy_upstream_reply_seconds_sum|cilium_services_events_total|cilium_triggers_policy_update_call_duration_seconds_count|cilium_triggers_policy_update_call_duration_seconds_sum|cilium_unreachable_nodes|cilium_version|hubble_dns_queries_total|hubble_dns_response_types_total|hubble_dns_responses_total|hubble_drop_total|hubble_flows_processed_total|hubble_http_request_duration_seconds_bucket|hubble_http_requests_total|hubble_http_responses_total|hubble_icmp_total|hubble_port_distribution_total|hubble_tcp_flags_total|isovalent_external_dns_proxy_policy_l7_total|isovalent_external_dns_proxy_processing_duration_seconds|isovalent_external_dns_proxy_update_errors_total|isovalent_external_dns_proxy_update_queue_size|timescape_clickhouse_queries_duration_seconds_bucket|timescape_clickhouse_queries_results_count|timescape_clickhouse_queries_results_sum|timescape_ingestor_flows_ingested_total|timescape_ingestor_ingest_duration_seconds_bucket|timescape_ingestor_ingest_running|timescape_ingestor_ingestfilter_batch_duration_seconds_bucket|timescape_ingestor_ingestfilter_filtered_errors_total|timescape_ingestor_ingestfilter_filtered_skipped_total|timescape_ingestor_ingestfilter_filtered_total|timescape_ingestor_ingestlog_getinfo_queries"
action = "keep"
}
}
prometheus.relabel "metrics_integrations_integrations_cilium_enterprise_hubble_timescape_server" {
forward_to = [prometheus.remote_write.metrics_service.receiver]
rule {
source_labels = ["__name__"]
regex = "cilium_agent_api_process_time_seconds_count|cilium_agent_api_process_time_seconds_sum|cilium_api_limiter_processed_requests_total|cilium_bpf_map_ops_total|cilium_bpf_map_pressure|cilium_controllers_runs_duration_seconds_count|cilium_controllers_runs_duration_seconds_sum|cilium_controllers_runs_total|cilium_datapath_conntrack_gc_duration_seconds_count|cilium_datapath_conntrack_gc_duration_seconds_sum|cilium_datapath_conntrack_gc_entries|cilium_datapath_conntrack_gc_key_fallbacks_total|cilium_datapath_conntrack_gc_runs_total|cilium_drop_bytes_total|cilium_drop_count_total|cilium_endpoint_regeneration_time_stats_seconds_count|cilium_endpoint_regeneration_time_stats_seconds_sum|cilium_endpoint_regenerations_total|cilium_endpoint_state|cilium_errors_warnings_total|cilium_forward_bytes_total|cilium_forward_count_total|cilium_identity|cilium_ip_addresses|cilium_k8s_client_api_calls_total|cilium_k8s_client_api_latency_time_seconds_count|cilium_k8s_client_api_latency_time_seconds_sum|cilium_kubernetes_events_received_total|cilium_kubernetes_events_total|cilium_nodes_all_events_received_total|cilium_nodes_all_num|cilium_operator_ces_queueing_delay_seconds_bucket|cilium_operator_ces_sync_errors_total|cilium_operator_ec2_api_duration_seconds_bucket|cilium_operator_identity_gc_entries|cilium_operator_identity_gc_runs|cilium_operator_ipam_allocation_ops|cilium_operator_ipam_deficit_resolver_duration_seconds_bucket|cilium_operator_ipam_interface_creation_ops|cilium_operator_ipam_ips|cilium_operator_ipam_k8s_sync_queued_total|cilium_operator_ipam_nodes|cilium_operator_ipam_resync_queued_total|cilium_operator_ipam_resync_total|cilium_operator_number_of_ceps_per_ces_sum|cilium_operator_process_cpu_seconds_total|cilium_operator_process_open_fds|cilium_operator_process_resident_memory_bytes|cilium_operator_process_virtual_memory_bytes|cilium_policy|cilium_policy_endpoint_enforcement_status|cilium_policy_l7_denied_total|cilium_policy_l7_forwarded_total|cilium_policy_l7_received_total|cilium_proxy_redirects|cilium_proxy_upstream_reply_seconds_count|cilium_proxy_upstream_reply_seconds_sum|cilium_services_events_total|cilium_triggers_policy_update_call_duration_seconds_count|cilium_triggers_policy_update_call_duration_seconds_sum|cilium_unreachable_nodes|cilium_version|hubble_dns_queries_total|hubble_dns_response_types_total|hubble_dns_responses_total|hubble_drop_total|hubble_flows_processed_total|hubble_http_request_duration_seconds_bucket|hubble_http_requests_total|hubble_http_responses_total|hubble_icmp_total|hubble_port_distribution_total|hubble_tcp_flags_total|isovalent_external_dns_proxy_policy_l7_total|isovalent_external_dns_proxy_processing_duration_seconds|isovalent_external_dns_proxy_update_errors_total|isovalent_external_dns_proxy_update_queue_size|timescape_clickhouse_queries_duration_seconds_bucket|timescape_clickhouse_queries_results_count|timescape_clickhouse_queries_results_sum|timescape_ingestor_flows_ingested_total|timescape_ingestor_ingest_duration_seconds_bucket|timescape_ingestor_ingest_running|timescape_ingestor_ingestfilter_batch_duration_seconds_bucket|timescape_ingestor_ingestfilter_filtered_errors_total|timescape_ingestor_ingestfilter_filtered_skipped_total|timescape_ingestor_ingestfilter_filtered_total|timescape_ingestor_ingestlog_getinfo_queries"
action = "keep"
}
}
prometheus.relabel "metrics_integrations_integrations_cilium_enterprise_cilium_agent" {
forward_to = [prometheus.remote_write.metrics_service.receiver]
rule {
source_labels = ["__name__"]
regex = "cilium_agent_api_process_time_seconds_count|cilium_agent_api_process_time_seconds_sum|cilium_api_limiter_processed_requests_total|cilium_bpf_map_ops_total|cilium_bpf_map_pressure|cilium_controllers_runs_duration_seconds_count|cilium_controllers_runs_duration_seconds_sum|cilium_controllers_runs_total|cilium_datapath_conntrack_gc_duration_seconds_count|cilium_datapath_conntrack_gc_duration_seconds_sum|cilium_datapath_conntrack_gc_entries|cilium_datapath_conntrack_gc_key_fallbacks_total|cilium_datapath_conntrack_gc_runs_total|cilium_drop_bytes_total|cilium_drop_count_total|cilium_endpoint_regeneration_time_stats_seconds_count|cilium_endpoint_regeneration_time_stats_seconds_sum|cilium_endpoint_regenerations_total|cilium_endpoint_state|cilium_errors_warnings_total|cilium_forward_bytes_total|cilium_forward_count_total|cilium_identity|cilium_ip_addresses|cilium_k8s_client_api_calls_total|cilium_k8s_client_api_latency_time_seconds_count|cilium_k8s_client_api_latency_time_seconds_sum|cilium_kubernetes_events_received_total|cilium_kubernetes_events_total|cilium_nodes_all_events_received_total|cilium_nodes_all_num|cilium_operator_ces_queueing_delay_seconds_bucket|cilium_operator_ces_sync_errors_total|cilium_operator_ec2_api_duration_seconds_bucket|cilium_operator_identity_gc_entries|cilium_operator_identity_gc_runs|cilium_operator_ipam_allocation_ops|cilium_operator_ipam_deficit_resolver_duration_seconds_bucket|cilium_operator_ipam_interface_creation_ops|cilium_operator_ipam_ips|cilium_operator_ipam_k8s_sync_queued_total|cilium_operator_ipam_nodes|cilium_operator_ipam_resync_queued_total|cilium_operator_ipam_resync_total|cilium_operator_number_of_ceps_per_ces_sum|cilium_operator_process_cpu_seconds_total|cilium_operator_process_open_fds|cilium_operator_process_resident_memory_bytes|cilium_operator_process_virtual_memory_bytes|cilium_policy|cilium_policy_endpoint_enforcement_status|cilium_policy_l7_denied_total|cilium_policy_l7_forwarded_total|cilium_policy_l7_received_total|cilium_proxy_redirects|cilium_proxy_upstream_reply_seconds_count|cilium_proxy_upstream_reply_seconds_sum|cilium_services_events_total|cilium_triggers_policy_update_call_duration_seconds_count|cilium_triggers_policy_update_call_duration_seconds_sum|cilium_unreachable_nodes|cilium_version|hubble_dns_queries_total|hubble_dns_response_types_total|hubble_dns_responses_total|hubble_drop_total|hubble_flows_processed_total|hubble_http_request_duration_seconds_bucket|hubble_http_requests_total|hubble_http_responses_total|hubble_icmp_total|hubble_port_distribution_total|hubble_tcp_flags_total|isovalent_external_dns_proxy_policy_l7_total|isovalent_external_dns_proxy_processing_duration_seconds|isovalent_external_dns_proxy_update_errors_total|isovalent_external_dns_proxy_update_queue_size|timescape_clickhouse_queries_duration_seconds_bucket|timescape_clickhouse_queries_results_count|timescape_clickhouse_queries_results_sum|timescape_ingestor_flows_ingested_total|timescape_ingestor_ingest_duration_seconds_bucket|timescape_ingestor_ingest_running|timescape_ingestor_ingestfilter_batch_duration_seconds_bucket|timescape_ingestor_ingestfilter_filtered_errors_total|timescape_ingestor_ingestfilter_filtered_skipped_total|timescape_ingestor_ingestfilter_filtered_total|timescape_ingestor_ingestlog_getinfo_queries"
action = "keep"
}
}
prometheus.relabel "metrics_integrations_integrations_cilium_enterprise_cilium_operator" {
forward_to = [prometheus.remote_write.metrics_service.receiver]
rule {
source_labels = ["__name__"]
regex = "cilium_agent_api_process_time_seconds_count|cilium_agent_api_process_time_seconds_sum|cilium_api_limiter_processed_requests_total|cilium_bpf_map_ops_total|cilium_bpf_map_pressure|cilium_controllers_runs_duration_seconds_count|cilium_controllers_runs_duration_seconds_sum|cilium_controllers_runs_total|cilium_datapath_conntrack_gc_duration_seconds_count|cilium_datapath_conntrack_gc_duration_seconds_sum|cilium_datapath_conntrack_gc_entries|cilium_datapath_conntrack_gc_key_fallbacks_total|cilium_datapath_conntrack_gc_runs_total|cilium_drop_bytes_total|cilium_drop_count_total|cilium_endpoint_regeneration_time_stats_seconds_count|cilium_endpoint_regeneration_time_stats_seconds_sum|cilium_endpoint_regenerations_total|cilium_endpoint_state|cilium_errors_warnings_total|cilium_forward_bytes_total|cilium_forward_count_total|cilium_identity|cilium_ip_addresses|cilium_k8s_client_api_calls_total|cilium_k8s_client_api_latency_time_seconds_count|cilium_k8s_client_api_latency_time_seconds_sum|cilium_kubernetes_events_received_total|cilium_kubernetes_events_total|cilium_nodes_all_events_received_total|cilium_nodes_all_num|cilium_operator_ces_queueing_delay_seconds_bucket|cilium_operator_ces_sync_errors_total|cilium_operator_ec2_api_duration_seconds_bucket|cilium_operator_identity_gc_entries|cilium_operator_identity_gc_runs|cilium_operator_ipam_allocation_ops|cilium_operator_ipam_deficit_resolver_duration_seconds_bucket|cilium_operator_ipam_interface_creation_ops|cilium_operator_ipam_ips|cilium_operator_ipam_k8s_sync_queued_total|cilium_operator_ipam_nodes|cilium_operator_ipam_resync_queued_total|cilium_operator_ipam_resync_total|cilium_operator_number_of_ceps_per_ces_sum|cilium_operator_process_cpu_seconds_total|cilium_operator_process_open_fds|cilium_operator_process_resident_memory_bytes|cilium_operator_process_virtual_memory_bytes|cilium_policy|cilium_policy_endpoint_enforcement_status|cilium_policy_l7_denied_total|cilium_policy_l7_forwarded_total|cilium_policy_l7_received_total|cilium_proxy_redirects|cilium_proxy_upstream_reply_seconds_count|cilium_proxy_upstream_reply_seconds_sum|cilium_services_events_total|cilium_triggers_policy_update_call_duration_seconds_count|cilium_triggers_policy_update_call_duration_seconds_sum|cilium_unreachable_nodes|cilium_version|hubble_dns_queries_total|hubble_dns_response_types_total|hubble_dns_responses_total|hubble_drop_total|hubble_flows_processed_total|hubble_http_request_duration_seconds_bucket|hubble_http_requests_total|hubble_http_responses_total|hubble_icmp_total|hubble_port_distribution_total|hubble_tcp_flags_total|isovalent_external_dns_proxy_policy_l7_total|isovalent_external_dns_proxy_processing_duration_seconds|isovalent_external_dns_proxy_update_errors_total|isovalent_external_dns_proxy_update_queue_size|timescape_clickhouse_queries_duration_seconds_bucket|timescape_clickhouse_queries_results_count|timescape_clickhouse_queries_results_sum|timescape_ingestor_flows_ingested_total|timescape_ingestor_ingest_duration_seconds_bucket|timescape_ingestor_ingest_running|timescape_ingestor_ingestfilter_batch_duration_seconds_bucket|timescape_ingestor_ingestfilter_filtered_errors_total|timescape_ingestor_ingestfilter_filtered_skipped_total|timescape_ingestor_ingestfilter_filtered_total|timescape_ingestor_ingestlog_getinfo_queries"
action = "keep"
}
}
prometheus.relabel "metrics_integrations_integrations_cilium_enterprise_hubble" {
forward_to = [prometheus.remote_write.metrics_service.receiver]
rule {
source_labels = ["__name__"]
regex = "cilium_agent_api_process_time_seconds_count|cilium_agent_api_process_time_seconds_sum|cilium_api_limiter_processed_requests_total|cilium_bpf_map_ops_total|cilium_bpf_map_pressure|cilium_controllers_runs_duration_seconds_count|cilium_controllers_runs_duration_seconds_sum|cilium_controllers_runs_total|cilium_datapath_conntrack_gc_duration_seconds_count|cilium_datapath_conntrack_gc_duration_seconds_sum|cilium_datapath_conntrack_gc_entries|cilium_datapath_conntrack_gc_key_fallbacks_total|cilium_datapath_conntrack_gc_runs_total|cilium_drop_bytes_total|cilium_drop_count_total|cilium_endpoint_regeneration_time_stats_seconds_count|cilium_endpoint_regeneration_time_stats_seconds_sum|cilium_endpoint_regenerations_total|cilium_endpoint_state|cilium_errors_warnings_total|cilium_forward_bytes_total|cilium_forward_count_total|cilium_identity|cilium_ip_addresses|cilium_k8s_client_api_calls_total|cilium_k8s_client_api_latency_time_seconds_count|cilium_k8s_client_api_latency_time_seconds_sum|cilium_kubernetes_events_received_total|cilium_kubernetes_events_total|cilium_nodes_all_events_received_total|cilium_nodes_all_num|cilium_operator_ces_queueing_delay_seconds_bucket|cilium_operator_ces_sync_errors_total|cilium_operator_ec2_api_duration_seconds_bucket|cilium_operator_identity_gc_entries|cilium_operator_identity_gc_runs|cilium_operator_ipam_allocation_ops|cilium_operator_ipam_deficit_resolver_duration_seconds_bucket|cilium_operator_ipam_interface_creation_ops|cilium_operator_ipam_ips|cilium_operator_ipam_k8s_sync_queued_total|cilium_operator_ipam_nodes|cilium_operator_ipam_resync_queued_total|cilium_operator_ipam_resync_total|cilium_operator_number_of_ceps_per_ces_sum|cilium_operator_process_cpu_seconds_total|cilium_operator_process_open_fds|cilium_operator_process_resident_memory_bytes|cilium_operator_process_virtual_memory_bytes|cilium_policy|cilium_policy_endpoint_enforcement_status|cilium_policy_l7_denied_total|cilium_policy_l7_forwarded_total|cilium_policy_l7_received_total|cilium_proxy_redirects|cilium_proxy_upstream_reply_seconds_count|cilium_proxy_upstream_reply_seconds_sum|cilium_services_events_total|cilium_triggers_policy_update_call_duration_seconds_count|cilium_triggers_policy_update_call_duration_seconds_sum|cilium_unreachable_nodes|cilium_version|hubble_dns_queries_total|hubble_dns_response_types_total|hubble_dns_responses_total|hubble_drop_total|hubble_flows_processed_total|hubble_http_request_duration_seconds_bucket|hubble_http_requests_total|hubble_http_responses_total|hubble_icmp_total|hubble_port_distribution_total|hubble_tcp_flags_total|isovalent_external_dns_proxy_policy_l7_total|isovalent_external_dns_proxy_processing_duration_seconds|isovalent_external_dns_proxy_update_errors_total|isovalent_external_dns_proxy_update_queue_size|timescape_clickhouse_queries_duration_seconds_bucket|timescape_clickhouse_queries_results_count|timescape_clickhouse_queries_results_sum|timescape_ingestor_flows_ingested_total|timescape_ingestor_ingest_duration_seconds_bucket|timescape_ingestor_ingest_running|timescape_ingestor_ingestfilter_batch_duration_seconds_bucket|timescape_ingestor_ingestfilter_filtered_errors_total|timescape_ingestor_ingestfilter_filtered_skipped_total|timescape_ingestor_ingestfilter_filtered_total|timescape_ingestor_ingestlog_getinfo_queries"
action = "keep"
}
}
prometheus.relabel "metrics_integrations_integrations_cilium_enterprise_hubble_enterprise" {
forward_to = [prometheus.remote_write.metrics_service.receiver]
rule {
source_labels = ["__name__"]
regex = "cilium_agent_api_process_time_seconds_count|cilium_agent_api_process_time_seconds_sum|cilium_api_limiter_processed_requests_total|cilium_bpf_map_ops_total|cilium_bpf_map_pressure|cilium_controllers_runs_duration_seconds_count|cilium_controllers_runs_duration_seconds_sum|cilium_controllers_runs_total|cilium_datapath_conntrack_gc_duration_seconds_count|cilium_datapath_conntrack_gc_duration_seconds_sum|cilium_datapath_conntrack_gc_entries|cilium_datapath_conntrack_gc_key_fallbacks_total|cilium_datapath_conntrack_gc_runs_total|cilium_drop_bytes_total|cilium_drop_count_total|cilium_endpoint_regeneration_time_stats_seconds_count|cilium_endpoint_regeneration_time_stats_seconds_sum|cilium_endpoint_regenerations_total|cilium_endpoint_state|cilium_errors_warnings_total|cilium_forward_bytes_total|cilium_forward_count_total|cilium_identity|cilium_ip_addresses|cilium_k8s_client_api_calls_total|cilium_k8s_client_api_latency_time_seconds_count|cilium_k8s_client_api_latency_time_seconds_sum|cilium_kubernetes_events_received_total|cilium_kubernetes_events_total|cilium_nodes_all_events_received_total|cilium_nodes_all_num|cilium_operator_ces_queueing_delay_seconds_bucket|cilium_operator_ces_sync_errors_total|cilium_operator_ec2_api_duration_seconds_bucket|cilium_operator_identity_gc_entries|cilium_operator_identity_gc_runs|cilium_operator_ipam_allocation_ops|cilium_operator_ipam_deficit_resolver_duration_seconds_bucket|cilium_operator_ipam_interface_creation_ops|cilium_operator_ipam_ips|cilium_operator_ipam_k8s_sync_queued_total|cilium_operator_ipam_nodes|cilium_operator_ipam_resync_queued_total|cilium_operator_ipam_resync_total|cilium_operator_number_of_ceps_per_ces_sum|cilium_operator_process_cpu_seconds_total|cilium_operator_process_open_fds|cilium_operator_process_resident_memory_bytes|cilium_operator_process_virtual_memory_bytes|cilium_policy|cilium_policy_endpoint_enforcement_status|cilium_policy_l7_denied_total|cilium_policy_l7_forwarded_total|cilium_policy_l7_received_total|cilium_proxy_redirects|cilium_proxy_upstream_reply_seconds_count|cilium_proxy_upstream_reply_seconds_sum|cilium_services_events_total|cilium_triggers_policy_update_call_duration_seconds_count|cilium_triggers_policy_update_call_duration_seconds_sum|cilium_unreachable_nodes|cilium_version|hubble_dns_queries_total|hubble_dns_response_types_total|hubble_dns_responses_total|hubble_drop_total|hubble_flows_processed_total|hubble_http_request_duration_seconds_bucket|hubble_http_requests_total|hubble_http_responses_total|hubble_icmp_total|hubble_port_distribution_total|hubble_tcp_flags_total|isovalent_external_dns_proxy_policy_l7_total|isovalent_external_dns_proxy_processing_duration_seconds|isovalent_external_dns_proxy_update_errors_total|isovalent_external_dns_proxy_update_queue_size|timescape_clickhouse_queries_duration_seconds_bucket|timescape_clickhouse_queries_results_count|timescape_clickhouse_queries_results_sum|timescape_ingestor_flows_ingested_total|timescape_ingestor_ingest_duration_seconds_bucket|timescape_ingestor_ingest_running|timescape_ingestor_ingestfilter_batch_duration_seconds_bucket|timescape_ingestor_ingestfilter_filtered_errors_total|timescape_ingestor_ingestfilter_filtered_skipped_total|timescape_ingestor_ingestfilter_filtered_total|timescape_ingestor_ingestlog_getinfo_queries"
action = "keep"
}
}
Advanced mode
The following snippets provide examples to guide you through the configuration process.
To instruct Grafana Alloy to scrape your Cilium Enterprise instances, manually copy and append the snippets to your alloy configuration file, then follow subsequent instructions.
Advanced metrics snippets
discovery.kubernetes "metrics_integrations_integrations_cilium_enterprise_hubble_timescape_ingestor" {
role = "endpoints"
}
discovery.kubernetes "metrics_integrations_integrations_cilium_enterprise_hubble_timescape_server" {
role = "endpoints"
}
discovery.kubernetes "metrics_integrations_integrations_cilium_enterprise_cilium_agent" {
role = "endpoints"
}
discovery.kubernetes "metrics_integrations_integrations_cilium_enterprise_cilium_operator" {
role = "endpoints"
}
discovery.kubernetes "metrics_integrations_integrations_cilium_enterprise_hubble" {
role = "endpoints"
}
discovery.kubernetes "metrics_integrations_integrations_cilium_enterprise_hubble_enterprise" {
role = "endpoints"
}
discovery.relabel "metrics_integrations_integrations_cilium_enterprise_hubble_timescape_ingestor" {
targets = discovery.kubernetes.metrics_integrations_integrations_cilium_enterprise_hubble_timescape_ingestor.targets
rule {
source_labels = ["__meta_kubernetes_service_label_app_kubernetes_io_instance", "__meta_kubernetes_service_labelpresent_app_kubernetes_io_instance"]
regex = "(hubble-timescape);true"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_service_label_app_kubernetes_io_name", "__meta_kubernetes_service_labelpresent_app_kubernetes_io_name"]
regex = "(hubble-timescape-ingestor);true"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_port_name"]
regex = "metrics"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_address_target_kind", "__meta_kubernetes_endpoint_address_target_name"]
regex = "Node;(.*)"
target_label = "node"
replacement = "${1}"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_address_target_kind", "__meta_kubernetes_endpoint_address_target_name"]
regex = "Pod;(.*)"
target_label = "pod"
replacement = "${1}"
}
rule {
source_labels = ["__meta_kubernetes_namespace"]
target_label = "namespace"
}
rule {
source_labels = ["__meta_kubernetes_service_name"]
target_label = "service"
}
rule {
source_labels = ["__meta_kubernetes_pod_name"]
target_label = "pod"
}
rule {
source_labels = ["__meta_kubernetes_pod_container_name"]
target_label = "container"
}
rule {
target_label = "endpoint"
replacement = "metrics"
}
rule {
source_labels = ["__address__"]
modulus = 1
target_label = "__tmp_hash"
action = "hashmod"
}
rule {
source_labels = ["__tmp_hash"]
regex = "0"
action = "keep"
}
}
discovery.relabel "metrics_integrations_integrations_cilium_enterprise_hubble_timescape_server" {
targets = discovery.kubernetes.metrics_integrations_integrations_cilium_enterprise_hubble_timescape_server.targets
rule {
source_labels = ["__meta_kubernetes_service_label_app_kubernetes_io_instance", "__meta_kubernetes_service_labelpresent_app_kubernetes_io_instance"]
regex = "(hubble-timescape);true"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_service_label_app_kubernetes_io_name", "__meta_kubernetes_service_labelpresent_app_kubernetes_io_name"]
regex = "(hubble-timescape-server);true"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_port_name"]
regex = "metrics"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_address_target_kind", "__meta_kubernetes_endpoint_address_target_name"]
regex = "Node;(.*)"
target_label = "node"
replacement = "${1}"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_address_target_kind", "__meta_kubernetes_endpoint_address_target_name"]
regex = "Pod;(.*)"
target_label = "pod"
replacement = "${1}"
}
rule {
source_labels = ["__meta_kubernetes_namespace"]
target_label = "namespace"
}
rule {
source_labels = ["__meta_kubernetes_service_name"]
target_label = "service"
}
rule {
source_labels = ["__meta_kubernetes_pod_name"]
target_label = "pod"
}
rule {
source_labels = ["__meta_kubernetes_pod_container_name"]
target_label = "container"
}
rule {
target_label = "endpoint"
replacement = "metrics"
}
rule {
source_labels = ["__address__"]
modulus = 1
target_label = "__tmp_hash"
action = "hashmod"
}
rule {
source_labels = ["__tmp_hash"]
regex = "0"
action = "keep"
}
}
discovery.relabel "metrics_integrations_integrations_cilium_enterprise_cilium_agent" {
targets = discovery.kubernetes.metrics_integrations_integrations_cilium_enterprise_cilium_agent.targets
rule {
source_labels = ["__meta_kubernetes_service_label_k8s_app", "__meta_kubernetes_service_labelpresent_k8s_app"]
regex = "(cilium);true"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_port_name"]
regex = "metrics"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_address_target_kind", "__meta_kubernetes_endpoint_address_target_name"]
regex = "Node;(.*)"
target_label = "node"
replacement = "${1}"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_address_target_kind", "__meta_kubernetes_endpoint_address_target_name"]
regex = "Pod;(.*)"
target_label = "pod"
replacement = "${1}"
}
rule {
source_labels = ["__meta_kubernetes_namespace"]
target_label = "namespace"
}
rule {
source_labels = ["__meta_kubernetes_service_name"]
target_label = "service"
}
rule {
source_labels = ["__meta_kubernetes_pod_name"]
target_label = "pod"
}
rule {
source_labels = ["__meta_kubernetes_pod_container_name"]
target_label = "container"
}
rule {
source_labels = ["__meta_kubernetes_service_label_k8s_app"]
regex = "(.+)"
target_label = "k8s_app"
replacement = "${1}"
}
rule {
target_label = "endpoint"
replacement = "metrics"
}
rule {
source_labels = ["__address__"]
modulus = 1
target_label = "__tmp_hash"
action = "hashmod"
}
rule {
source_labels = ["__tmp_hash"]
regex = "0"
action = "keep"
}
}
discovery.relabel "metrics_integrations_integrations_cilium_enterprise_cilium_operator" {
targets = discovery.kubernetes.metrics_integrations_integrations_cilium_enterprise_cilium_operator.targets
rule {
source_labels = ["__meta_kubernetes_service_label_io_cilium_app", "__meta_kubernetes_service_labelpresent_io_cilium_app"]
regex = "(operator);true"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_service_label_name", "__meta_kubernetes_service_labelpresent_name"]
regex = "(cilium-operator);true"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_port_name"]
regex = "metrics"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_address_target_kind", "__meta_kubernetes_endpoint_address_target_name"]
regex = "Node;(.*)"
target_label = "node"
replacement = "${1}"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_address_target_kind", "__meta_kubernetes_endpoint_address_target_name"]
regex = "Pod;(.*)"
target_label = "pod"
replacement = "${1}"
}
rule {
source_labels = ["__meta_kubernetes_namespace"]
target_label = "namespace"
}
rule {
source_labels = ["__meta_kubernetes_service_name"]
target_label = "service"
}
rule {
source_labels = ["__meta_kubernetes_pod_name"]
target_label = "pod"
}
rule {
source_labels = ["__meta_kubernetes_pod_container_name"]
target_label = "container"
}
rule {
source_labels = ["__meta_kubernetes_service_label_io_cilium_app"]
regex = "(.+)"
target_label = "io_cilium_app"
replacement = "${1}"
}
rule {
target_label = "endpoint"
replacement = "metrics"
}
rule {
source_labels = ["__address__"]
modulus = 1
target_label = "__tmp_hash"
action = "hashmod"
}
rule {
source_labels = ["__tmp_hash"]
regex = "0"
action = "keep"
}
}
discovery.relabel "metrics_integrations_integrations_cilium_enterprise_hubble" {
targets = discovery.kubernetes.metrics_integrations_integrations_cilium_enterprise_hubble.targets
rule {
source_labels = ["__meta_kubernetes_service_label_k8s_app", "__meta_kubernetes_service_labelpresent_k8s_app"]
regex = "(hubble);true"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_port_name"]
regex = "hubble-metrics"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_address_target_kind", "__meta_kubernetes_endpoint_address_target_name"]
regex = "Node;(.*)"
target_label = "node"
replacement = "${1}"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_address_target_kind", "__meta_kubernetes_endpoint_address_target_name"]
regex = "Pod;(.*)"
target_label = "pod"
replacement = "${1}"
}
rule {
source_labels = ["__meta_kubernetes_namespace"]
target_label = "namespace"
}
rule {
source_labels = ["__meta_kubernetes_service_name"]
target_label = "service"
}
rule {
source_labels = ["__meta_kubernetes_pod_name"]
target_label = "pod"
}
rule {
source_labels = ["__meta_kubernetes_pod_container_name"]
target_label = "container"
}
rule {
target_label = "endpoint"
replacement = "hubble-metrics"
}
rule {
source_labels = ["__address__"]
modulus = 1
target_label = "__tmp_hash"
action = "hashmod"
}
rule {
source_labels = ["__tmp_hash"]
regex = "0"
action = "keep"
}
}
discovery.relabel "metrics_integrations_integrations_cilium_enterprise_hubble_enterprise" {
targets = discovery.kubernetes.metrics_integrations_integrations_cilium_enterprise_hubble_enterprise.targets
rule {
source_labels = ["__meta_kubernetes_service_label_app_kubernetes_io_instance", "__meta_kubernetes_service_labelpresent_app_kubernetes_io_instance"]
regex = "(hubble-enterprise);true"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_service_label_app_kubernetes_io_managed_by", "__meta_kubernetes_service_labelpresent_app_kubernetes_io_managed_by"]
regex = "(Helm);true"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_service_label_app_kubernetes_io_name", "__meta_kubernetes_service_labelpresent_app_kubernetes_io_name"]
regex = "(hubble-enterprise);true"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_service_label_helm_sh_chart", "__meta_kubernetes_service_labelpresent_helm_sh_chart"]
regex = "(hubble-enterprise-9999.9999.9999-dev);true"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_port_name"]
regex = "metrics"
action = "keep"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_address_target_kind", "__meta_kubernetes_endpoint_address_target_name"]
regex = "Node;(.*)"
target_label = "node"
replacement = "${1}"
}
rule {
source_labels = ["__meta_kubernetes_endpoint_address_target_kind", "__meta_kubernetes_endpoint_address_target_name"]
regex = "Pod;(.*)"
target_label = "pod"
replacement = "${1}"
}
rule {
source_labels = ["__meta_kubernetes_namespace"]
target_label = "namespace"
}
rule {
source_labels = ["__meta_kubernetes_service_name"]
target_label = "service"
}
rule {
source_labels = ["__meta_kubernetes_pod_name"]
target_label = "pod"
}
rule {
source_labels = ["__meta_kubernetes_pod_container_name"]
target_label = "container"
}
rule {
target_label = "endpoint"
replacement = "metrics"
}
rule {
source_labels = ["__meta_kubernetes_pod_node_name"]
target_label = "node"
replacement = "${1}"
}
rule {
source_labels = ["__address__"]
modulus = 1
target_label = "__tmp_hash"
action = "hashmod"
}
rule {
source_labels = ["__tmp_hash"]
regex = "0"
action = "keep"
}
}
prometheus.scrape "metrics_integrations_integrations_cilium_enterprise_hubble_timescape_ingestor" {
targets = discovery.relabel.metrics_integrations_integrations_cilium_enterprise_hubble_timescape_ingestor.output
forward_to = [prometheus.relabel.metrics_integrations_integrations_cilium_enterprise_hubble_timescape_ingestor.receiver]
job_name = "integrations/cilium-enterprise/hubble-timescape-ingestor"
honor_labels = true
scrape_timeout = "1m0s"
}
prometheus.scrape "metrics_integrations_integrations_cilium_enterprise_hubble_timescape_server" {
targets = discovery.relabel.metrics_integrations_integrations_cilium_enterprise_hubble_timescape_server.output
forward_to = [prometheus.relabel.metrics_integrations_integrations_cilium_enterprise_hubble_timescape_server.receiver]
job_name = "integrations/cilium-enterprise/hubble-timescape-server"
honor_labels = true
scrape_timeout = "1m0s"
}
prometheus.scrape "metrics_integrations_integrations_cilium_enterprise_cilium_agent" {
targets = discovery.relabel.metrics_integrations_integrations_cilium_enterprise_cilium_agent.output
forward_to = [prometheus.relabel.metrics_integrations_integrations_cilium_enterprise_cilium_agent.receiver]
job_name = "integrations/cilium-enterprise/cilium-agent"
honor_labels = true
scrape_timeout = "1m0s"
}
prometheus.scrape "metrics_integrations_integrations_cilium_enterprise_cilium_operator" {
targets = discovery.relabel.metrics_integrations_integrations_cilium_enterprise_cilium_operator.output
forward_to = [prometheus.relabel.metrics_integrations_integrations_cilium_enterprise_cilium_operator.receiver]
job_name = "integrations/cilium-enterprise/cilium-operator"
honor_labels = true
scrape_timeout = "1m0s"
}
prometheus.scrape "metrics_integrations_integrations_cilium_enterprise_hubble" {
targets = discovery.relabel.metrics_integrations_integrations_cilium_enterprise_hubble.output
forward_to = [prometheus.relabel.metrics_integrations_integrations_cilium_enterprise_hubble.receiver]
job_name = "integrations/cilium-enterprise/hubble"
honor_labels = true
scrape_timeout = "1m0s"
}
prometheus.scrape "metrics_integrations_integrations_cilium_enterprise_hubble_enterprise" {
targets = discovery.relabel.metrics_integrations_integrations_cilium_enterprise_hubble_enterprise.output
forward_to = [prometheus.relabel.metrics_integrations_integrations_cilium_enterprise_hubble_enterprise.receiver]
job_name = "integrations/cilium-enterprise/hubble-enterprise"
honor_labels = true
scrape_timeout = "1m0s"
}
prometheus.relabel "metrics_integrations_integrations_cilium_enterprise_hubble_timescape_ingestor" {
forward_to = [prometheus.remote_write.metrics_service.receiver]
rule {
source_labels = ["__name__"]
regex = "cilium_agent_api_process_time_seconds_count|cilium_agent_api_process_time_seconds_sum|cilium_api_limiter_processed_requests_total|cilium_bpf_map_ops_total|cilium_bpf_map_pressure|cilium_controllers_runs_duration_seconds_count|cilium_controllers_runs_duration_seconds_sum|cilium_controllers_runs_total|cilium_datapath_conntrack_gc_duration_seconds_count|cilium_datapath_conntrack_gc_duration_seconds_sum|cilium_datapath_conntrack_gc_entries|cilium_datapath_conntrack_gc_key_fallbacks_total|cilium_datapath_conntrack_gc_runs_total|cilium_drop_bytes_total|cilium_drop_count_total|cilium_endpoint_regeneration_time_stats_seconds_count|cilium_endpoint_regeneration_time_stats_seconds_sum|cilium_endpoint_regenerations_total|cilium_endpoint_state|cilium_errors_warnings_total|cilium_forward_bytes_total|cilium_forward_count_total|cilium_identity|cilium_ip_addresses|cilium_k8s_client_api_calls_total|cilium_k8s_client_api_latency_time_seconds_count|cilium_k8s_client_api_latency_time_seconds_sum|cilium_kubernetes_events_received_total|cilium_kubernetes_events_total|cilium_nodes_all_events_received_total|cilium_nodes_all_num|cilium_operator_ces_queueing_delay_seconds_bucket|cilium_operator_ces_sync_errors_total|cilium_operator_ec2_api_duration_seconds_bucket|cilium_operator_identity_gc_entries|cilium_operator_identity_gc_runs|cilium_operator_ipam_allocation_ops|cilium_operator_ipam_deficit_resolver_duration_seconds_bucket|cilium_operator_ipam_interface_creation_ops|cilium_operator_ipam_ips|cilium_operator_ipam_k8s_sync_queued_total|cilium_operator_ipam_nodes|cilium_operator_ipam_resync_queued_total|cilium_operator_ipam_resync_total|cilium_operator_number_of_ceps_per_ces_sum|cilium_operator_process_cpu_seconds_total|cilium_operator_process_open_fds|cilium_operator_process_resident_memory_bytes|cilium_operator_process_virtual_memory_bytes|cilium_policy|cilium_policy_endpoint_enforcement_status|cilium_policy_l7_denied_total|cilium_policy_l7_forwarded_total|cilium_policy_l7_received_total|cilium_proxy_redirects|cilium_proxy_upstream_reply_seconds_count|cilium_proxy_upstream_reply_seconds_sum|cilium_services_events_total|cilium_triggers_policy_update_call_duration_seconds_count|cilium_triggers_policy_update_call_duration_seconds_sum|cilium_unreachable_nodes|cilium_version|hubble_dns_queries_total|hubble_dns_response_types_total|hubble_dns_responses_total|hubble_drop_total|hubble_flows_processed_total|hubble_http_request_duration_seconds_bucket|hubble_http_requests_total|hubble_http_responses_total|hubble_icmp_total|hubble_port_distribution_total|hubble_tcp_flags_total|isovalent_external_dns_proxy_policy_l7_total|isovalent_external_dns_proxy_processing_duration_seconds|isovalent_external_dns_proxy_update_errors_total|isovalent_external_dns_proxy_update_queue_size|timescape_clickhouse_queries_duration_seconds_bucket|timescape_clickhouse_queries_results_count|timescape_clickhouse_queries_results_sum|timescape_ingestor_flows_ingested_total|timescape_ingestor_ingest_duration_seconds_bucket|timescape_ingestor_ingest_running|timescape_ingestor_ingestfilter_batch_duration_seconds_bucket|timescape_ingestor_ingestfilter_filtered_errors_total|timescape_ingestor_ingestfilter_filtered_skipped_total|timescape_ingestor_ingestfilter_filtered_total|timescape_ingestor_ingestlog_getinfo_queries"
action = "keep"
}
}
prometheus.relabel "metrics_integrations_integrations_cilium_enterprise_hubble_timescape_server" {
forward_to = [prometheus.remote_write.metrics_service.receiver]
rule {
source_labels = ["__name__"]
regex = "cilium_agent_api_process_time_seconds_count|cilium_agent_api_process_time_seconds_sum|cilium_api_limiter_processed_requests_total|cilium_bpf_map_ops_total|cilium_bpf_map_pressure|cilium_controllers_runs_duration_seconds_count|cilium_controllers_runs_duration_seconds_sum|cilium_controllers_runs_total|cilium_datapath_conntrack_gc_duration_seconds_count|cilium_datapath_conntrack_gc_duration_seconds_sum|cilium_datapath_conntrack_gc_entries|cilium_datapath_conntrack_gc_key_fallbacks_total|cilium_datapath_conntrack_gc_runs_total|cilium_drop_bytes_total|cilium_drop_count_total|cilium_endpoint_regeneration_time_stats_seconds_count|cilium_endpoint_regeneration_time_stats_seconds_sum|cilium_endpoint_regenerations_total|cilium_endpoint_state|cilium_errors_warnings_total|cilium_forward_bytes_total|cilium_forward_count_total|cilium_identity|cilium_ip_addresses|cilium_k8s_client_api_calls_total|cilium_k8s_client_api_latency_time_seconds_count|cilium_k8s_client_api_latency_time_seconds_sum|cilium_kubernetes_events_received_total|cilium_kubernetes_events_total|cilium_nodes_all_events_received_total|cilium_nodes_all_num|cilium_operator_ces_queueing_delay_seconds_bucket|cilium_operator_ces_sync_errors_total|cilium_operator_ec2_api_duration_seconds_bucket|cilium_operator_identity_gc_entries|cilium_operator_identity_gc_runs|cilium_operator_ipam_allocation_ops|cilium_operator_ipam_deficit_resolver_duration_seconds_bucket|cilium_operator_ipam_interface_creation_ops|cilium_operator_ipam_ips|cilium_operator_ipam_k8s_sync_queued_total|cilium_operator_ipam_nodes|cilium_operator_ipam_resync_queued_total|cilium_operator_ipam_resync_total|cilium_operator_number_of_ceps_per_ces_sum|cilium_operator_process_cpu_seconds_total|cilium_operator_process_open_fds|cilium_operator_process_resident_memory_bytes|cilium_operator_process_virtual_memory_bytes|cilium_policy|cilium_policy_endpoint_enforcement_status|cilium_policy_l7_denied_total|cilium_policy_l7_forwarded_total|cilium_policy_l7_received_total|cilium_proxy_redirects|cilium_proxy_upstream_reply_seconds_count|cilium_proxy_upstream_reply_seconds_sum|cilium_services_events_total|cilium_triggers_policy_update_call_duration_seconds_count|cilium_triggers_policy_update_call_duration_seconds_sum|cilium_unreachable_nodes|cilium_version|hubble_dns_queries_total|hubble_dns_response_types_total|hubble_dns_responses_total|hubble_drop_total|hubble_flows_processed_total|hubble_http_request_duration_seconds_bucket|hubble_http_requests_total|hubble_http_responses_total|hubble_icmp_total|hubble_port_distribution_total|hubble_tcp_flags_total|isovalent_external_dns_proxy_policy_l7_total|isovalent_external_dns_proxy_processing_duration_seconds|isovalent_external_dns_proxy_update_errors_total|isovalent_external_dns_proxy_update_queue_size|timescape_clickhouse_queries_duration_seconds_bucket|timescape_clickhouse_queries_results_count|timescape_clickhouse_queries_results_sum|timescape_ingestor_flows_ingested_total|timescape_ingestor_ingest_duration_seconds_bucket|timescape_ingestor_ingest_running|timescape_ingestor_ingestfilter_batch_duration_seconds_bucket|timescape_ingestor_ingestfilter_filtered_errors_total|timescape_ingestor_ingestfilter_filtered_skipped_total|timescape_ingestor_ingestfilter_filtered_total|timescape_ingestor_ingestlog_getinfo_queries"
action = "keep"
}
}
prometheus.relabel "metrics_integrations_integrations_cilium_enterprise_cilium_agent" {
forward_to = [prometheus.remote_write.metrics_service.receiver]
rule {
source_labels = ["__name__"]
regex = "cilium_agent_api_process_time_seconds_count|cilium_agent_api_process_time_seconds_sum|cilium_api_limiter_processed_requests_total|cilium_bpf_map_ops_total|cilium_bpf_map_pressure|cilium_controllers_runs_duration_seconds_count|cilium_controllers_runs_duration_seconds_sum|cilium_controllers_runs_total|cilium_datapath_conntrack_gc_duration_seconds_count|cilium_datapath_conntrack_gc_duration_seconds_sum|cilium_datapath_conntrack_gc_entries|cilium_datapath_conntrack_gc_key_fallbacks_total|cilium_datapath_conntrack_gc_runs_total|cilium_drop_bytes_total|cilium_drop_count_total|cilium_endpoint_regeneration_time_stats_seconds_count|cilium_endpoint_regeneration_time_stats_seconds_sum|cilium_endpoint_regenerations_total|cilium_endpoint_state|cilium_errors_warnings_total|cilium_forward_bytes_total|cilium_forward_count_total|cilium_identity|cilium_ip_addresses|cilium_k8s_client_api_calls_total|cilium_k8s_client_api_latency_time_seconds_count|cilium_k8s_client_api_latency_time_seconds_sum|cilium_kubernetes_events_received_total|cilium_kubernetes_events_total|cilium_nodes_all_events_received_total|cilium_nodes_all_num|cilium_operator_ces_queueing_delay_seconds_bucket|cilium_operator_ces_sync_errors_total|cilium_operator_ec2_api_duration_seconds_bucket|cilium_operator_identity_gc_entries|cilium_operator_identity_gc_runs|cilium_operator_ipam_allocation_ops|cilium_operator_ipam_deficit_resolver_duration_seconds_bucket|cilium_operator_ipam_interface_creation_ops|cilium_operator_ipam_ips|cilium_operator_ipam_k8s_sync_queued_total|cilium_operator_ipam_nodes|cilium_operator_ipam_resync_queued_total|cilium_operator_ipam_resync_total|cilium_operator_number_of_ceps_per_ces_sum|cilium_operator_process_cpu_seconds_total|cilium_operator_process_open_fds|cilium_operator_process_resident_memory_bytes|cilium_operator_process_virtual_memory_bytes|cilium_policy|cilium_policy_endpoint_enforcement_status|cilium_policy_l7_denied_total|cilium_policy_l7_forwarded_total|cilium_policy_l7_received_total|cilium_proxy_redirects|cilium_proxy_upstream_reply_seconds_count|cilium_proxy_upstream_reply_seconds_sum|cilium_services_events_total|cilium_triggers_policy_update_call_duration_seconds_count|cilium_triggers_policy_update_call_duration_seconds_sum|cilium_unreachable_nodes|cilium_version|hubble_dns_queries_total|hubble_dns_response_types_total|hubble_dns_responses_total|hubble_drop_total|hubble_flows_processed_total|hubble_http_request_duration_seconds_bucket|hubble_http_requests_total|hubble_http_responses_total|hubble_icmp_total|hubble_port_distribution_total|hubble_tcp_flags_total|isovalent_external_dns_proxy_policy_l7_total|isovalent_external_dns_proxy_processing_duration_seconds|isovalent_external_dns_proxy_update_errors_total|isovalent_external_dns_proxy_update_queue_size|timescape_clickhouse_queries_duration_seconds_bucket|timescape_clickhouse_queries_results_count|timescape_clickhouse_queries_results_sum|timescape_ingestor_flows_ingested_total|timescape_ingestor_ingest_duration_seconds_bucket|timescape_ingestor_ingest_running|timescape_ingestor_ingestfilter_batch_duration_seconds_bucket|timescape_ingestor_ingestfilter_filtered_errors_total|timescape_ingestor_ingestfilter_filtered_skipped_total|timescape_ingestor_ingestfilter_filtered_total|timescape_ingestor_ingestlog_getinfo_queries"
action = "keep"
}
}
prometheus.relabel "metrics_integrations_integrations_cilium_enterprise_cilium_operator" {
forward_to = [prometheus.remote_write.metrics_service.receiver]
rule {
source_labels = ["__name__"]
regex = "cilium_agent_api_process_time_seconds_count|cilium_agent_api_process_time_seconds_sum|cilium_api_limiter_processed_requests_total|cilium_bpf_map_ops_total|cilium_bpf_map_pressure|cilium_controllers_runs_duration_seconds_count|cilium_controllers_runs_duration_seconds_sum|cilium_controllers_runs_total|cilium_datapath_conntrack_gc_duration_seconds_count|cilium_datapath_conntrack_gc_duration_seconds_sum|cilium_datapath_conntrack_gc_entries|cilium_datapath_conntrack_gc_key_fallbacks_total|cilium_datapath_conntrack_gc_runs_total|cilium_drop_bytes_total|cilium_drop_count_total|cilium_endpoint_regeneration_time_stats_seconds_count|cilium_endpoint_regeneration_time_stats_seconds_sum|cilium_endpoint_regenerations_total|cilium_endpoint_state|cilium_errors_warnings_total|cilium_forward_bytes_total|cilium_forward_count_total|cilium_identity|cilium_ip_addresses|cilium_k8s_client_api_calls_total|cilium_k8s_client_api_latency_time_seconds_count|cilium_k8s_client_api_latency_time_seconds_sum|cilium_kubernetes_events_received_total|cilium_kubernetes_events_total|cilium_nodes_all_events_received_total|cilium_nodes_all_num|cilium_operator_ces_queueing_delay_seconds_bucket|cilium_operator_ces_sync_errors_total|cilium_operator_ec2_api_duration_seconds_bucket|cilium_operator_identity_gc_entries|cilium_operator_identity_gc_runs|cilium_operator_ipam_allocation_ops|cilium_operator_ipam_deficit_resolver_duration_seconds_bucket|cilium_operator_ipam_interface_creation_ops|cilium_operator_ipam_ips|cilium_operator_ipam_k8s_sync_queued_total|cilium_operator_ipam_nodes|cilium_operator_ipam_resync_queued_total|cilium_operator_ipam_resync_total|cilium_operator_number_of_ceps_per_ces_sum|cilium_operator_process_cpu_seconds_total|cilium_operator_process_open_fds|cilium_operator_process_resident_memory_bytes|cilium_operator_process_virtual_memory_bytes|cilium_policy|cilium_policy_endpoint_enforcement_status|cilium_policy_l7_denied_total|cilium_policy_l7_forwarded_total|cilium_policy_l7_received_total|cilium_proxy_redirects|cilium_proxy_upstream_reply_seconds_count|cilium_proxy_upstream_reply_seconds_sum|cilium_services_events_total|cilium_triggers_policy_update_call_duration_seconds_count|cilium_triggers_policy_update_call_duration_seconds_sum|cilium_unreachable_nodes|cilium_version|hubble_dns_queries_total|hubble_dns_response_types_total|hubble_dns_responses_total|hubble_drop_total|hubble_flows_processed_total|hubble_http_request_duration_seconds_bucket|hubble_http_requests_total|hubble_http_responses_total|hubble_icmp_total|hubble_port_distribution_total|hubble_tcp_flags_total|isovalent_external_dns_proxy_policy_l7_total|isovalent_external_dns_proxy_processing_duration_seconds|isovalent_external_dns_proxy_update_errors_total|isovalent_external_dns_proxy_update_queue_size|timescape_clickhouse_queries_duration_seconds_bucket|timescape_clickhouse_queries_results_count|timescape_clickhouse_queries_results_sum|timescape_ingestor_flows_ingested_total|timescape_ingestor_ingest_duration_seconds_bucket|timescape_ingestor_ingest_running|timescape_ingestor_ingestfilter_batch_duration_seconds_bucket|timescape_ingestor_ingestfilter_filtered_errors_total|timescape_ingestor_ingestfilter_filtered_skipped_total|timescape_ingestor_ingestfilter_filtered_total|timescape_ingestor_ingestlog_getinfo_queries"
action = "keep"
}
}
prometheus.relabel "metrics_integrations_integrations_cilium_enterprise_hubble" {
forward_to = [prometheus.remote_write.metrics_service.receiver]
rule {
source_labels = ["__name__"]
regex = "cilium_agent_api_process_time_seconds_count|cilium_agent_api_process_time_seconds_sum|cilium_api_limiter_processed_requests_total|cilium_bpf_map_ops_total|cilium_bpf_map_pressure|cilium_controllers_runs_duration_seconds_count|cilium_controllers_runs_duration_seconds_sum|cilium_controllers_runs_total|cilium_datapath_conntrack_gc_duration_seconds_count|cilium_datapath_conntrack_gc_duration_seconds_sum|cilium_datapath_conntrack_gc_entries|cilium_datapath_conntrack_gc_key_fallbacks_total|cilium_datapath_conntrack_gc_runs_total|cilium_drop_bytes_total|cilium_drop_count_total|cilium_endpoint_regeneration_time_stats_seconds_count|cilium_endpoint_regeneration_time_stats_seconds_sum|cilium_endpoint_regenerations_total|cilium_endpoint_state|cilium_errors_warnings_total|cilium_forward_bytes_total|cilium_forward_count_total|cilium_identity|cilium_ip_addresses|cilium_k8s_client_api_calls_total|cilium_k8s_client_api_latency_time_seconds_count|cilium_k8s_client_api_latency_time_seconds_sum|cilium_kubernetes_events_received_total|cilium_kubernetes_events_total|cilium_nodes_all_events_received_total|cilium_nodes_all_num|cilium_operator_ces_queueing_delay_seconds_bucket|cilium_operator_ces_sync_errors_total|cilium_operator_ec2_api_duration_seconds_bucket|cilium_operator_identity_gc_entries|cilium_operator_identity_gc_runs|cilium_operator_ipam_allocation_ops|cilium_operator_ipam_deficit_resolver_duration_seconds_bucket|cilium_operator_ipam_interface_creation_ops|cilium_operator_ipam_ips|cilium_operator_ipam_k8s_sync_queued_total|cilium_operator_ipam_nodes|cilium_operator_ipam_resync_queued_total|cilium_operator_ipam_resync_total|cilium_operator_number_of_ceps_per_ces_sum|cilium_operator_process_cpu_seconds_total|cilium_operator_process_open_fds|cilium_operator_process_resident_memory_bytes|cilium_operator_process_virtual_memory_bytes|cilium_policy|cilium_policy_endpoint_enforcement_status|cilium_policy_l7_denied_total|cilium_policy_l7_forwarded_total|cilium_policy_l7_received_total|cilium_proxy_redirects|cilium_proxy_upstream_reply_seconds_count|cilium_proxy_upstream_reply_seconds_sum|cilium_services_events_total|cilium_triggers_policy_update_call_duration_seconds_count|cilium_triggers_policy_update_call_duration_seconds_sum|cilium_unreachable_nodes|cilium_version|hubble_dns_queries_total|hubble_dns_response_types_total|hubble_dns_responses_total|hubble_drop_total|hubble_flows_processed_total|hubble_http_request_duration_seconds_bucket|hubble_http_requests_total|hubble_http_responses_total|hubble_icmp_total|hubble_port_distribution_total|hubble_tcp_flags_total|isovalent_external_dns_proxy_policy_l7_total|isovalent_external_dns_proxy_processing_duration_seconds|isovalent_external_dns_proxy_update_errors_total|isovalent_external_dns_proxy_update_queue_size|timescape_clickhouse_queries_duration_seconds_bucket|timescape_clickhouse_queries_results_count|timescape_clickhouse_queries_results_sum|timescape_ingestor_flows_ingested_total|timescape_ingestor_ingest_duration_seconds_bucket|timescape_ingestor_ingest_running|timescape_ingestor_ingestfilter_batch_duration_seconds_bucket|timescape_ingestor_ingestfilter_filtered_errors_total|timescape_ingestor_ingestfilter_filtered_skipped_total|timescape_ingestor_ingestfilter_filtered_total|timescape_ingestor_ingestlog_getinfo_queries"
action = "keep"
}
}
prometheus.relabel "metrics_integrations_integrations_cilium_enterprise_hubble_enterprise" {
forward_to = [prometheus.remote_write.metrics_service.receiver]
rule {
source_labels = ["__name__"]
regex = "cilium_agent_api_process_time_seconds_count|cilium_agent_api_process_time_seconds_sum|cilium_api_limiter_processed_requests_total|cilium_bpf_map_ops_total|cilium_bpf_map_pressure|cilium_controllers_runs_duration_seconds_count|cilium_controllers_runs_duration_seconds_sum|cilium_controllers_runs_total|cilium_datapath_conntrack_gc_duration_seconds_count|cilium_datapath_conntrack_gc_duration_seconds_sum|cilium_datapath_conntrack_gc_entries|cilium_datapath_conntrack_gc_key_fallbacks_total|cilium_datapath_conntrack_gc_runs_total|cilium_drop_bytes_total|cilium_drop_count_total|cilium_endpoint_regeneration_time_stats_seconds_count|cilium_endpoint_regeneration_time_stats_seconds_sum|cilium_endpoint_regenerations_total|cilium_endpoint_state|cilium_errors_warnings_total|cilium_forward_bytes_total|cilium_forward_count_total|cilium_identity|cilium_ip_addresses|cilium_k8s_client_api_calls_total|cilium_k8s_client_api_latency_time_seconds_count|cilium_k8s_client_api_latency_time_seconds_sum|cilium_kubernetes_events_received_total|cilium_kubernetes_events_total|cilium_nodes_all_events_received_total|cilium_nodes_all_num|cilium_operator_ces_queueing_delay_seconds_bucket|cilium_operator_ces_sync_errors_total|cilium_operator_ec2_api_duration_seconds_bucket|cilium_operator_identity_gc_entries|cilium_operator_identity_gc_runs|cilium_operator_ipam_allocation_ops|cilium_operator_ipam_deficit_resolver_duration_seconds_bucket|cilium_operator_ipam_interface_creation_ops|cilium_operator_ipam_ips|cilium_operator_ipam_k8s_sync_queued_total|cilium_operator_ipam_nodes|cilium_operator_ipam_resync_queued_total|cilium_operator_ipam_resync_total|cilium_operator_number_of_ceps_per_ces_sum|cilium_operator_process_cpu_seconds_total|cilium_operator_process_open_fds|cilium_operator_process_resident_memory_bytes|cilium_operator_process_virtual_memory_bytes|cilium_policy|cilium_policy_endpoint_enforcement_status|cilium_policy_l7_denied_total|cilium_policy_l7_forwarded_total|cilium_policy_l7_received_total|cilium_proxy_redirects|cilium_proxy_upstream_reply_seconds_count|cilium_proxy_upstream_reply_seconds_sum|cilium_services_events_total|cilium_triggers_policy_update_call_duration_seconds_count|cilium_triggers_policy_update_call_duration_seconds_sum|cilium_unreachable_nodes|cilium_version|hubble_dns_queries_total|hubble_dns_response_types_total|hubble_dns_responses_total|hubble_drop_total|hubble_flows_processed_total|hubble_http_request_duration_seconds_bucket|hubble_http_requests_total|hubble_http_responses_total|hubble_icmp_total|hubble_port_distribution_total|hubble_tcp_flags_total|isovalent_external_dns_proxy_policy_l7_total|isovalent_external_dns_proxy_processing_duration_seconds|isovalent_external_dns_proxy_update_errors_total|isovalent_external_dns_proxy_update_queue_size|timescape_clickhouse_queries_duration_seconds_bucket|timescape_clickhouse_queries_results_count|timescape_clickhouse_queries_results_sum|timescape_ingestor_flows_ingested_total|timescape_ingestor_ingest_duration_seconds_bucket|timescape_ingestor_ingest_running|timescape_ingestor_ingestfilter_batch_duration_seconds_bucket|timescape_ingestor_ingestfilter_filtered_errors_total|timescape_ingestor_ingestfilter_filtered_skipped_total|timescape_ingestor_ingestfilter_filtered_total|timescape_ingestor_ingestlog_getinfo_queries"
action = "keep"
}
}
To monitor your Cilium Enterprise instance, you must use a discovery.relabel component to discover your Cilium Enterprise Prometheus endpoint and apply appropriate labels, followed by a prometheus.scrape component to scrape it.
Configure the following properties within each discovery.relabel
component:
__address__
: The address to your Cilium Enterprise Prometheus metrics endpoint.instance
label:constants.hostname
sets theinstance
label to your Grafana Alloy server hostname. If that is not suitable, change it to a value uniquely identifies this Cilium Enterprise instance.
If you have multiple Cilium Enterprise servers to scrape, configure one discovery.relabel
for each and scrape them by including each under targets
within the prometheus.scrape
component.
Grafana Agent static configuration (deprecated)
The following section shows configuration for running Grafana Agent in static mode which is deprecated. You should use Grafana Alloy for all new deployments.
Before you begin
Please note, this integration is intended exclusively for use with Cilium Enterprise, and does not support the Cilium OSS product.
This integration monitors a Cilium Enterprise & Hubble Enterprise deployment that has metrics exporters enabled. Please ensure you have completed the following setup steps:
- Enabled the embedded Prometheus exporter in your Cilium deployment to collect and expose metrics
- Enabled the embedded Prometheus exporter in Hubble if you want Hubble metrics to be included.
Once the exporters have been enabled the metrics are automatically exposed and available to be scraped by either Prometheus or a Grafana Agent deployed to the cluster.
The following sample using Helm enables the Prometheus metrics endpoint and configures the relevant metrics for Cilium Agent, Cilium Operator, and Hubble:
helm install cilium cilium/cilium --version 1.12.2 \
--namespace kube-system \
--set prometheus.enabled=true \
--set operator.prometheus.enabled=true
Sample Hubble metrics setup using helm, specifying which metrics to enable capture and export for:
helm install cilium cilium/cilium --version 1.12.2 \
--namespace kube-system \
--set hubble.metrics.enabled="{dns,drop,tcp,flow,icmp,http}"
Install Cilium Enterprise integration for Grafana Cloud
- In your Grafana Cloud stack, click Connections in the left-hand menu.
- Find Cilium Enterprise and click its tile to open the integration.
- Review the prerequisites in the Configuration Details tab and set up Grafana Agent to send Cilium Enterprise metrics to your Grafana Cloud instance.
- Click Install to add this integration’s pre-built dashboards and alerts to your Grafana Cloud instance, and you can start monitoring your Cilium Enterprise setup.
Post-install configuration for the Cilium Enterprise integration
Prerequisites
Before you begin, you should have the following available:
- A Kubernetes cluster with role-based access control (RBAC) enabled.
- The
kubectl
command-line tool installed on your local machine, configured to connect to your cluster. To learn more aboutkubectl
, see the Kubernetes documentation.
Configure Grafana Agent
The Cilium Enterprise integration uses the Grafana Agent, deployed into your cluster, to scrape metrics from your Cilium and Hubble deployment.
The agent must be configured to scrape your Cilium and Hubble deployment correctly. To do so, please be sure to replace the capitalized variables beginning with YOUR_*
with the appropriate cluster name and Grafana Cloud credentials.
Please note: If configuring multiple clusters to report metrics, ensure unique cluster names are used!
Paste the following script into your shell and run it to configure the Grafana Agent:
cat <<'EOF' |
kind: ConfigMap
metadata:
name: grafana-agent
apiVersion: v1
data:
agent.yaml: |
metrics:
wal_directory: /var/lib/agent/wal
global:
scrape_interval: 60s
external_labels:
cluster: YOUR_CLUSTER_NAME
configs:
- name: integrations
remote_write:
- url: YOUR_PROMETHEUS_REMOTE_WRITE_URL
basic_auth:
username: YOUR_PROMETHEUS_REMOTE_WRITE_USERNAME
password: YOUR_PROMETHEUS_REMOTE_WRITE_PASSWORD
scrape_configs:
- job_name: integrations/cilium-enterprise/hubble-timescape-ingestor
honor_labels: true
honor_timestamps: true
scrape_interval: 60s
scrape_timeout: 60s
metrics_path: /metrics
scheme: http
follow_redirects: true
metric_relabel_configs:
- source_labels: [__name__]
regex: cilium_controllers_runs_duration_seconds_count|hubble_http_requests_total|cilium_operator_process_resident_memory_bytes|cilium_k8s_client_api_calls_total|cilium_controllers_runs_duration_seconds_sum|cilium_operator_ipam_nodes|cilium_datapath_conntrack_gc_entries|hubble_flows_processed_total|cilium_proxy_upstream_reply_seconds_count|timescape_ingestor_ingest_running|cilium_policy_l7_denied_total|timescape_ingestor_ingestfilter_filtered_skipped_total|cilium_operator_process_cpu_seconds_total|cilium_operator_ec2_api_duration_seconds_sum|cilium_controllers_runs_total|timescape_ingestor_ingestfilter_filtered_total|cilium_policy_l7_received_total|hubble_http_responses_total|cilium_version|timescape_clickhouse_queries_results_count|cilium_drop_bytes_total|cilium_ip_addresses|cilium_operator_ipam_interface_creation_ops|cilium_operator_ec2_api_rate_limit_duration_seconds_sum|cilium_k8s_client_api_latency_time_seconds_count|cilium_forward_bytes_total|isovalent_external_dns_proxy_processing_duration_seconds|cilium_services_events_total|cilium_endpoint_regeneration_time_stats_seconds_count|cilium_kubernetes_events_total|cilium_triggers_policy_update_call_duration_seconds_sum|cilium_triggers_policy_update_call_duration_seconds_count|timescape_ingestor_ingestfilter_filtered_errors_total|timescape_clickhouse_queries_results_sum|isovalent_external_dns_proxy_update_queue_size|cilium_operator_ec2_api_duration_seconds_count|cilium_operator_ec2_api_rate_limit_duration_seconds_count|timescape_ingestor_ingestlog_getinfo_queries|hubble_tcp_flags_total|hubble_icmp_total|cilium_policy_endpoint_enforcement_status|cilium_forward_count_total|cilium_datapath_conntrack_gc_duration_seconds_sum|hubble_dns_response_types_total|isovalent_external_dns_proxy_update_errors_total|cilium_operator_ipam_resync_total|cilium_drop_count_total|hubble_dns_queries_total|cilium_policy_l7_forwarded_total|cilium_agent_api_process_time_seconds_count|timescape_clickhouse_queries_duration_seconds_bucket|timescape_ingestor_ingestfilter_batch_duration_seconds_bucket|cilium_endpoint_regenerations_total|cilium_proxy_upstream_reply_seconds_sum|cilium_identity|cilium_k8s_client_api_latency_time_seconds_sum|cilium_bpf_map_ops_total|cilium_errors_warnings_total|cilium_datapath_conntrack_gc_duration_seconds_count|hubble_dns_responses_total|cilium_operator_ipam_ips|cilium_endpoint_state|cilium_endpoint_regeneration_time_stats_seconds_sum|hubble_port_distribution_total|cilium_nodes_all_num|isovalent_external_dns_proxy_policy_l7_total|cilium_datapath_conntrack_gc_runs_total|timescape_ingestor_ingest_duration_seconds_bucket|hubble_drop_total|cilium_agent_api_process_time_seconds_sum|timescape_ingestor_flows_ingested_total|cilium_proxy_redirects|cilium_datapath_conntrack_gc_key_fallbacks_total|hubble_http_request_duration_seconds_bucket|cilium_nodes_all_events_received_total|cilium_kubernetes_events_received_total|cilium_policy|cilium_operator_ipam_available
action: keep
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_instance, __meta_kubernetes_service_labelpresent_app_kubernetes_io_instance]
separator: ;
regex: (hubble-timescape);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name, __meta_kubernetes_service_labelpresent_app_kubernetes_io_name]
separator: ;
regex: (hubble-timescape-ingestor);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: metrics
action: replace
- source_labels: [__address__]
separator: ;
regex: (.*)
modulus: 1
target_label: __tmp_hash
replacement: $1
action: hashmod
- source_labels: [__tmp_hash]
separator: ;
regex: "0"
replacement: $1
action: keep
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
follow_redirects: true
- job_name: integrations/cilium-enterprise/hubble-timescape-server
honor_labels: true
honor_timestamps: true
scrape_interval: 60s
scrape_timeout: 60s
metrics_path: /metrics
scheme: http
follow_redirects: true
metric_relabel_configs:
- source_labels: [__name__]
regex: cilium_controllers_runs_duration_seconds_count|hubble_http_requests_total|cilium_operator_process_resident_memory_bytes|cilium_k8s_client_api_calls_total|cilium_controllers_runs_duration_seconds_sum|cilium_operator_ipam_nodes|cilium_datapath_conntrack_gc_entries|hubble_flows_processed_total|cilium_proxy_upstream_reply_seconds_count|timescape_ingestor_ingest_running|cilium_policy_l7_denied_total|timescape_ingestor_ingestfilter_filtered_skipped_total|cilium_operator_process_cpu_seconds_total|cilium_operator_ec2_api_duration_seconds_sum|cilium_controllers_runs_total|timescape_ingestor_ingestfilter_filtered_total|cilium_policy_l7_received_total|hubble_http_responses_total|cilium_version|timescape_clickhouse_queries_results_count|cilium_drop_bytes_total|cilium_ip_addresses|cilium_operator_ipam_interface_creation_ops|cilium_operator_ec2_api_rate_limit_duration_seconds_sum|cilium_k8s_client_api_latency_time_seconds_count|cilium_forward_bytes_total|isovalent_external_dns_proxy_processing_duration_seconds|cilium_services_events_total|cilium_endpoint_regeneration_time_stats_seconds_count|cilium_kubernetes_events_total|cilium_triggers_policy_update_call_duration_seconds_sum|cilium_triggers_policy_update_call_duration_seconds_count|timescape_ingestor_ingestfilter_filtered_errors_total|timescape_clickhouse_queries_results_sum|isovalent_external_dns_proxy_update_queue_size|cilium_operator_ec2_api_duration_seconds_count|cilium_operator_ec2_api_rate_limit_duration_seconds_count|timescape_ingestor_ingestlog_getinfo_queries|hubble_tcp_flags_total|hubble_icmp_total|cilium_policy_endpoint_enforcement_status|cilium_forward_count_total|cilium_datapath_conntrack_gc_duration_seconds_sum|hubble_dns_response_types_total|isovalent_external_dns_proxy_update_errors_total|cilium_operator_ipam_resync_total|cilium_drop_count_total|hubble_dns_queries_total|cilium_policy_l7_forwarded_total|cilium_agent_api_process_time_seconds_count|timescape_clickhouse_queries_duration_seconds_bucket|timescape_ingestor_ingestfilter_batch_duration_seconds_bucket|cilium_endpoint_regenerations_total|cilium_proxy_upstream_reply_seconds_sum|cilium_identity|cilium_k8s_client_api_latency_time_seconds_sum|cilium_bpf_map_ops_total|cilium_errors_warnings_total|cilium_datapath_conntrack_gc_duration_seconds_count|hubble_dns_responses_total|cilium_operator_ipam_ips|cilium_endpoint_state|cilium_endpoint_regeneration_time_stats_seconds_sum|hubble_port_distribution_total|cilium_nodes_all_num|isovalent_external_dns_proxy_policy_l7_total|cilium_datapath_conntrack_gc_runs_total|timescape_ingestor_ingest_duration_seconds_bucket|hubble_drop_total|cilium_agent_api_process_time_seconds_sum|timescape_ingestor_flows_ingested_total|cilium_proxy_redirects|cilium_datapath_conntrack_gc_key_fallbacks_total|hubble_http_request_duration_seconds_bucket|cilium_nodes_all_events_received_total|cilium_kubernetes_events_received_total|cilium_policy|cilium_operator_ipam_available
action: keep
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_instance, __meta_kubernetes_service_labelpresent_app_kubernetes_io_instance]
separator: ;
regex: (hubble-timescape);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name, __meta_kubernetes_service_labelpresent_app_kubernetes_io_name]
separator: ;
regex: (hubble-timescape-server);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: metrics
action: replace
- source_labels: [__address__]
separator: ;
regex: (.*)
modulus: 1
target_label: __tmp_hash
replacement: $1
action: hashmod
- source_labels: [__tmp_hash]
separator: ;
regex: "0"
replacement: $1
action: keep
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
follow_redirects: true
- job_name: integrations/cilium-enterprise/cilium-agent
honor_labels: true
honor_timestamps: true
scrape_interval: 60s
scrape_timeout: 60s
metrics_path: /metrics
scheme: http
follow_redirects: true
metric_relabel_configs:
- source_labels: [__name__]
regex: cilium_controllers_runs_duration_seconds_count|hubble_http_requests_total|cilium_operator_process_resident_memory_bytes|cilium_k8s_client_api_calls_total|cilium_controllers_runs_duration_seconds_sum|cilium_operator_ipam_nodes|cilium_datapath_conntrack_gc_entries|hubble_flows_processed_total|cilium_proxy_upstream_reply_seconds_count|timescape_ingestor_ingest_running|cilium_policy_l7_denied_total|timescape_ingestor_ingestfilter_filtered_skipped_total|cilium_operator_process_cpu_seconds_total|cilium_operator_ec2_api_duration_seconds_sum|cilium_controllers_runs_total|timescape_ingestor_ingestfilter_filtered_total|cilium_policy_l7_received_total|hubble_http_responses_total|cilium_version|timescape_clickhouse_queries_results_count|cilium_drop_bytes_total|cilium_ip_addresses|cilium_operator_ipam_interface_creation_ops|cilium_operator_ec2_api_rate_limit_duration_seconds_sum|cilium_k8s_client_api_latency_time_seconds_count|cilium_forward_bytes_total|isovalent_external_dns_proxy_processing_duration_seconds|cilium_services_events_total|cilium_endpoint_regeneration_time_stats_seconds_count|cilium_kubernetes_events_total|cilium_triggers_policy_update_call_duration_seconds_sum|cilium_triggers_policy_update_call_duration_seconds_count|timescape_ingestor_ingestfilter_filtered_errors_total|timescape_clickhouse_queries_results_sum|isovalent_external_dns_proxy_update_queue_size|cilium_operator_ec2_api_duration_seconds_count|cilium_operator_ec2_api_rate_limit_duration_seconds_count|timescape_ingestor_ingestlog_getinfo_queries|hubble_tcp_flags_total|hubble_icmp_total|cilium_policy_endpoint_enforcement_status|cilium_forward_count_total|cilium_datapath_conntrack_gc_duration_seconds_sum|hubble_dns_response_types_total|isovalent_external_dns_proxy_update_errors_total|cilium_operator_ipam_resync_total|cilium_drop_count_total|hubble_dns_queries_total|cilium_policy_l7_forwarded_total|cilium_agent_api_process_time_seconds_count|timescape_clickhouse_queries_duration_seconds_bucket|timescape_ingestor_ingestfilter_batch_duration_seconds_bucket|cilium_endpoint_regenerations_total|cilium_proxy_upstream_reply_seconds_sum|cilium_identity|cilium_k8s_client_api_latency_time_seconds_sum|cilium_bpf_map_ops_total|cilium_errors_warnings_total|cilium_datapath_conntrack_gc_duration_seconds_count|hubble_dns_responses_total|cilium_operator_ipam_ips|cilium_endpoint_state|cilium_endpoint_regeneration_time_stats_seconds_sum|hubble_port_distribution_total|cilium_nodes_all_num|isovalent_external_dns_proxy_policy_l7_total|cilium_datapath_conntrack_gc_runs_total|timescape_ingestor_ingest_duration_seconds_bucket|hubble_drop_total|cilium_agent_api_process_time_seconds_sum|timescape_ingestor_flows_ingested_total|cilium_proxy_redirects|cilium_datapath_conntrack_gc_key_fallbacks_total|hubble_http_request_duration_seconds_bucket|cilium_nodes_all_events_received_total|cilium_kubernetes_events_received_total|cilium_policy|cilium_operator_ipam_available
action: keep
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_k8s_app, __meta_kubernetes_service_labelpresent_k8s_app]
separator: ;
regex: (cilium);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_label_k8s_app]
separator: ;
regex: (.+)
target_label: k8s_app
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: metrics
action: replace
- source_labels: [__address__]
separator: ;
regex: (.*)
modulus: 1
target_label: __tmp_hash
replacement: $1
action: hashmod
- source_labels: [__tmp_hash]
separator: ;
regex: "0"
replacement: $1
action: keep
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
follow_redirects: true
- job_name: integrations/cilium-enterprise/cilium-operator
honor_labels: true
honor_timestamps: true
scrape_interval: 60s
scrape_timeout: 60s
metrics_path: /metrics
scheme: http
follow_redirects: true
metric_relabel_configs:
- source_labels: [__name__]
regex: cilium_controllers_runs_duration_seconds_count|hubble_http_requests_total|cilium_operator_process_resident_memory_bytes|cilium_k8s_client_api_calls_total|cilium_controllers_runs_duration_seconds_sum|cilium_operator_ipam_nodes|cilium_datapath_conntrack_gc_entries|hubble_flows_processed_total|cilium_proxy_upstream_reply_seconds_count|timescape_ingestor_ingest_running|cilium_policy_l7_denied_total|timescape_ingestor_ingestfilter_filtered_skipped_total|cilium_operator_process_cpu_seconds_total|cilium_operator_ec2_api_duration_seconds_sum|cilium_controllers_runs_total|timescape_ingestor_ingestfilter_filtered_total|cilium_policy_l7_received_total|hubble_http_responses_total|cilium_version|timescape_clickhouse_queries_results_count|cilium_drop_bytes_total|cilium_ip_addresses|cilium_operator_ipam_interface_creation_ops|cilium_operator_ec2_api_rate_limit_duration_seconds_sum|cilium_k8s_client_api_latency_time_seconds_count|cilium_forward_bytes_total|isovalent_external_dns_proxy_processing_duration_seconds|cilium_services_events_total|cilium_endpoint_regeneration_time_stats_seconds_count|cilium_kubernetes_events_total|cilium_triggers_policy_update_call_duration_seconds_sum|cilium_triggers_policy_update_call_duration_seconds_count|timescape_ingestor_ingestfilter_filtered_errors_total|timescape_clickhouse_queries_results_sum|isovalent_external_dns_proxy_update_queue_size|cilium_operator_ec2_api_duration_seconds_count|cilium_operator_ec2_api_rate_limit_duration_seconds_count|timescape_ingestor_ingestlog_getinfo_queries|hubble_tcp_flags_total|hubble_icmp_total|cilium_policy_endpoint_enforcement_status|cilium_forward_count_total|cilium_datapath_conntrack_gc_duration_seconds_sum|hubble_dns_response_types_total|isovalent_external_dns_proxy_update_errors_total|cilium_operator_ipam_resync_total|cilium_drop_count_total|hubble_dns_queries_total|cilium_policy_l7_forwarded_total|cilium_agent_api_process_time_seconds_count|timescape_clickhouse_queries_duration_seconds_bucket|timescape_ingestor_ingestfilter_batch_duration_seconds_bucket|cilium_endpoint_regenerations_total|cilium_proxy_upstream_reply_seconds_sum|cilium_identity|cilium_k8s_client_api_latency_time_seconds_sum|cilium_bpf_map_ops_total|cilium_errors_warnings_total|cilium_datapath_conntrack_gc_duration_seconds_count|hubble_dns_responses_total|cilium_operator_ipam_ips|cilium_endpoint_state|cilium_endpoint_regeneration_time_stats_seconds_sum|hubble_port_distribution_total|cilium_nodes_all_num|isovalent_external_dns_proxy_policy_l7_total|cilium_datapath_conntrack_gc_runs_total|timescape_ingestor_ingest_duration_seconds_bucket|hubble_drop_total|cilium_agent_api_process_time_seconds_sum|timescape_ingestor_flows_ingested_total|cilium_proxy_redirects|cilium_datapath_conntrack_gc_key_fallbacks_total|hubble_http_request_duration_seconds_bucket|cilium_nodes_all_events_received_total|cilium_kubernetes_events_received_total|cilium_policy|cilium_operator_ipam_available
action: keep
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_io_cilium_app, __meta_kubernetes_service_labelpresent_io_cilium_app]
separator: ;
regex: (operator);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_service_label_name, __meta_kubernetes_service_labelpresent_name]
separator: ;
regex: (cilium-operator);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_label_io_cilium_app]
separator: ;
regex: (.+)
target_label: io_cilium_app
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: metrics
action: replace
- source_labels: [__address__]
separator: ;
regex: (.*)
modulus: 1
target_label: __tmp_hash
replacement: $1
action: hashmod
- source_labels: [__tmp_hash]
separator: ;
regex: "0"
replacement: $1
action: keep
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
follow_redirects: true
- job_name: integrations/cilium-enterprise/hubble
honor_labels: true
honor_timestamps: true
scrape_interval: 60s
scrape_timeout: 60s
metrics_path: /metrics
scheme: http
follow_redirects: true
metric_relabel_configs:
- source_labels: [__name__]
regex: cilium_controllers_runs_duration_seconds_count|hubble_http_requests_total|cilium_operator_process_resident_memory_bytes|cilium_k8s_client_api_calls_total|cilium_controllers_runs_duration_seconds_sum|cilium_operator_ipam_nodes|cilium_datapath_conntrack_gc_entries|hubble_flows_processed_total|cilium_proxy_upstream_reply_seconds_count|timescape_ingestor_ingest_running|cilium_policy_l7_denied_total|timescape_ingestor_ingestfilter_filtered_skipped_total|cilium_operator_process_cpu_seconds_total|cilium_operator_ec2_api_duration_seconds_sum|cilium_controllers_runs_total|timescape_ingestor_ingestfilter_filtered_total|cilium_policy_l7_received_total|hubble_http_responses_total|cilium_version|timescape_clickhouse_queries_results_count|cilium_drop_bytes_total|cilium_ip_addresses|cilium_operator_ipam_interface_creation_ops|cilium_operator_ec2_api_rate_limit_duration_seconds_sum|cilium_k8s_client_api_latency_time_seconds_count|cilium_forward_bytes_total|isovalent_external_dns_proxy_processing_duration_seconds|cilium_services_events_total|cilium_endpoint_regeneration_time_stats_seconds_count|cilium_kubernetes_events_total|cilium_triggers_policy_update_call_duration_seconds_sum|cilium_triggers_policy_update_call_duration_seconds_count|timescape_ingestor_ingestfilter_filtered_errors_total|timescape_clickhouse_queries_results_sum|isovalent_external_dns_proxy_update_queue_size|cilium_operator_ec2_api_duration_seconds_count|cilium_operator_ec2_api_rate_limit_duration_seconds_count|timescape_ingestor_ingestlog_getinfo_queries|hubble_tcp_flags_total|hubble_icmp_total|cilium_policy_endpoint_enforcement_status|cilium_forward_count_total|cilium_datapath_conntrack_gc_duration_seconds_sum|hubble_dns_response_types_total|isovalent_external_dns_proxy_update_errors_total|cilium_operator_ipam_resync_total|cilium_drop_count_total|hubble_dns_queries_total|cilium_policy_l7_forwarded_total|cilium_agent_api_process_time_seconds_count|timescape_clickhouse_queries_duration_seconds_bucket|timescape_ingestor_ingestfilter_batch_duration_seconds_bucket|cilium_endpoint_regenerations_total|cilium_proxy_upstream_reply_seconds_sum|cilium_identity|cilium_k8s_client_api_latency_time_seconds_sum|cilium_bpf_map_ops_total|cilium_errors_warnings_total|cilium_datapath_conntrack_gc_duration_seconds_count|hubble_dns_responses_total|cilium_operator_ipam_ips|cilium_endpoint_state|cilium_endpoint_regeneration_time_stats_seconds_sum|hubble_port_distribution_total|cilium_nodes_all_num|isovalent_external_dns_proxy_policy_l7_total|cilium_datapath_conntrack_gc_runs_total|timescape_ingestor_ingest_duration_seconds_bucket|hubble_drop_total|cilium_agent_api_process_time_seconds_sum|timescape_ingestor_flows_ingested_total|cilium_proxy_redirects|cilium_datapath_conntrack_gc_key_fallbacks_total|hubble_http_request_duration_seconds_bucket|cilium_nodes_all_events_received_total|cilium_kubernetes_events_received_total|cilium_policy|cilium_operator_ipam_available
action: keep
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_k8s_app, __meta_kubernetes_service_labelpresent_k8s_app]
separator: ;
regex: (hubble);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: hubble-metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: hubble-metrics
action: replace
- source_labels: [__address__]
separator: ;
regex: (.*)
modulus: 1
target_label: __tmp_hash
replacement: $1
action: hashmod
- source_labels: [__tmp_hash]
separator: ;
regex: "0"
replacement: $1
action: keep
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
follow_redirects: true
- job_name: integrations/cilium-enterprise/hubble-enterprise
honor_labels: true
honor_timestamps: true
scrape_interval: 60s
scrape_timeout: 60s
metrics_path: /metrics
scheme: http
follow_redirects: true
metric_relabel_configs:
- source_labels: [__name__]
regex: cilium_controllers_runs_duration_seconds_count|hubble_http_requests_total|cilium_operator_process_resident_memory_bytes|cilium_k8s_client_api_calls_total|cilium_controllers_runs_duration_seconds_sum|cilium_operator_ipam_nodes|cilium_datapath_conntrack_gc_entries|hubble_flows_processed_total|cilium_proxy_upstream_reply_seconds_count|timescape_ingestor_ingest_running|cilium_policy_l7_denied_total|timescape_ingestor_ingestfilter_filtered_skipped_total|cilium_operator_process_cpu_seconds_total|cilium_operator_ec2_api_duration_seconds_sum|cilium_controllers_runs_total|timescape_ingestor_ingestfilter_filtered_total|cilium_policy_l7_received_total|hubble_http_responses_total|cilium_version|timescape_clickhouse_queries_results_count|cilium_drop_bytes_total|cilium_ip_addresses|cilium_operator_ipam_interface_creation_ops|cilium_operator_ec2_api_rate_limit_duration_seconds_sum|cilium_k8s_client_api_latency_time_seconds_count|cilium_forward_bytes_total|isovalent_external_dns_proxy_processing_duration_seconds|cilium_services_events_total|cilium_endpoint_regeneration_time_stats_seconds_count|cilium_kubernetes_events_total|cilium_triggers_policy_update_call_duration_seconds_sum|cilium_triggers_policy_update_call_duration_seconds_count|timescape_ingestor_ingestfilter_filtered_errors_total|timescape_clickhouse_queries_results_sum|isovalent_external_dns_proxy_update_queue_size|cilium_operator_ec2_api_duration_seconds_count|cilium_operator_ec2_api_rate_limit_duration_seconds_count|timescape_ingestor_ingestlog_getinfo_queries|hubble_tcp_flags_total|hubble_icmp_total|cilium_policy_endpoint_enforcement_status|cilium_forward_count_total|cilium_datapath_conntrack_gc_duration_seconds_sum|hubble_dns_response_types_total|isovalent_external_dns_proxy_update_errors_total|cilium_operator_ipam_resync_total|cilium_drop_count_total|hubble_dns_queries_total|cilium_policy_l7_forwarded_total|cilium_agent_api_process_time_seconds_count|timescape_clickhouse_queries_duration_seconds_bucket|timescape_ingestor_ingestfilter_batch_duration_seconds_bucket|cilium_endpoint_regenerations_total|cilium_proxy_upstream_reply_seconds_sum|cilium_identity|cilium_k8s_client_api_latency_time_seconds_sum|cilium_bpf_map_ops_total|cilium_errors_warnings_total|cilium_datapath_conntrack_gc_duration_seconds_count|hubble_dns_responses_total|cilium_operator_ipam_ips|cilium_endpoint_state|cilium_endpoint_regeneration_time_stats_seconds_sum|hubble_port_distribution_total|cilium_nodes_all_num|isovalent_external_dns_proxy_policy_l7_total|cilium_datapath_conntrack_gc_runs_total|timescape_ingestor_ingest_duration_seconds_bucket|hubble_drop_total|cilium_agent_api_process_time_seconds_sum|timescape_ingestor_flows_ingested_total|cilium_proxy_redirects|cilium_datapath_conntrack_gc_key_fallbacks_total|hubble_http_request_duration_seconds_bucket|cilium_nodes_all_events_received_total|cilium_kubernetes_events_received_total|cilium_policy|cilium_operator_ipam_available
action: keep
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_instance, __meta_kubernetes_service_labelpresent_app_kubernetes_io_instance]
separator: ;
regex: (hubble-enterprise);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_managed_by,
__meta_kubernetes_service_labelpresent_app_kubernetes_io_managed_by]
separator: ;
regex: (Helm);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name, __meta_kubernetes_service_labelpresent_app_kubernetes_io_name]
separator: ;
regex: (hubble-enterprise);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_service_label_helm_sh_chart, __meta_kubernetes_service_labelpresent_helm_sh_chart]
separator: ;
regex: (hubble-enterprise-9999.9999.9999-dev);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: metrics
action: replace
- source_labels: [__meta_kubernetes_pod_node_name]
separator: ;
regex: (.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__address__]
separator: ;
regex: (.*)
modulus: 1
target_label: __tmp_hash
replacement: $1
action: hashmod
- source_labels: [__tmp_hash]
separator: ;
regex: "0"
replacement: $1
action: keep
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
follow_redirects: true
EOF
(export NAMESPACE=default && kubectl apply -n $NAMESPACE -f -)
If you deployed the Agent into a non-default Namespace in the previous step, replace NAMESPACE=default
in this command with the new Namespace.
This ConfigMap confiures the Agent to scape the Cilium Agent
, Cilium Operator
, Hubble
, and Hubble Timesscape
resources in your cluster and ship these scraped metrics to Grafana Cloud.
This ConfigMap configures the Agent to scrape the cadvisor
and kubelet
endpoints in your cluster and ship these scraped metrics to Grafana Cloud.
To learn more about configuring the Agent, please see Configure Grafana Agent from the Agent docs.
Deploy Grafana Agent resources
In this step you’ll install the Grafana Agent and its required resources into your cluster.
Run the following command from your shell to install the Grafana Agent into the default
Namespace of your Kubernetes cluster:
MANIFEST_URL=https://raw.githubusercontent.com/grafana/agent/v0.24.2/production/kubernetes/agent-bare.yaml NAMESPACE=default /bin/sh -c "$(curl -fsSL https://raw.githubusercontent.com/grafana/agent/v0.24.2/production/kubernetes/install-bare.sh)" | kubectl apply -f -
This installs a single replica Grafana Agent StatefulSet into your cluster and configures RBAC permissions for the Agent. If you would like to deploy the Agent into a different Namespace, change the NAMESPACE=default
variable, ensuring that this Namespace already exists.
Configuration snippets for Grafana Agent
Below metrics.configs.scrape_configs
, insert the following lines and change the URLs according to your environment:
- job_name: integrations/cilium-enterprise/hubble-timescape-ingestor
honor_labels: true
honor_timestamps: true
scrape_timeout: 60s
metrics_path: /metrics
scheme: http
follow_redirects: true
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_instance, __meta_kubernetes_service_labelpresent_app_kubernetes_io_instance]
separator: ;
regex: (hubble-timescape);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name, __meta_kubernetes_service_labelpresent_app_kubernetes_io_name]
separator: ;
regex: (hubble-timescape-ingestor);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: metrics
action: replace
- source_labels: [__address__]
separator: ;
regex: (.*)
modulus: 1
target_label: __tmp_hash
replacement: $1
action: hashmod
- source_labels: [__tmp_hash]
separator: ;
regex: "0"
replacement: $1
action: keep
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
follow_redirects: true
- job_name: integrations/cilium-enterprise/hubble-timescape-server
honor_labels: true
honor_timestamps: true
scrape_timeout: 60s
metrics_path: /metrics
scheme: http
follow_redirects: true
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_instance, __meta_kubernetes_service_labelpresent_app_kubernetes_io_instance]
separator: ;
regex: (hubble-timescape);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name, __meta_kubernetes_service_labelpresent_app_kubernetes_io_name]
separator: ;
regex: (hubble-timescape-server);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: metrics
action: replace
- source_labels: [__address__]
separator: ;
regex: (.*)
modulus: 1
target_label: __tmp_hash
replacement: $1
action: hashmod
- source_labels: [__tmp_hash]
separator: ;
regex: "0"
replacement: $1
action: keep
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
follow_redirects: true
- job_name: integrations/cilium-enterprise/cilium-agent
honor_labels: true
honor_timestamps: true
scrape_timeout: 60s
metrics_path: /metrics
scheme: http
follow_redirects: true
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_k8s_app, __meta_kubernetes_service_labelpresent_k8s_app]
separator: ;
regex: (cilium);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_label_k8s_app]
separator: ;
regex: (.+)
target_label: k8s_app
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: metrics
action: replace
- source_labels: [__address__]
separator: ;
regex: (.*)
modulus: 1
target_label: __tmp_hash
replacement: $1
action: hashmod
- source_labels: [__tmp_hash]
separator: ;
regex: "0"
replacement: $1
action: keep
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
follow_redirects: true
- job_name: integrations/cilium-enterprise/cilium-operator
honor_labels: true
honor_timestamps: true
scrape_timeout: 60s
metrics_path: /metrics
scheme: http
follow_redirects: true
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_io_cilium_app, __meta_kubernetes_service_labelpresent_io_cilium_app]
separator: ;
regex: (operator);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_service_label_name, __meta_kubernetes_service_labelpresent_name]
separator: ;
regex: (cilium-operator);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_label_io_cilium_app]
separator: ;
regex: (.+)
target_label: io_cilium_app
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: metrics
action: replace
- source_labels: [__address__]
separator: ;
regex: (.*)
modulus: 1
target_label: __tmp_hash
replacement: $1
action: hashmod
- source_labels: [__tmp_hash]
separator: ;
regex: "0"
replacement: $1
action: keep
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
follow_redirects: true
- job_name: integrations/cilium-enterprise/hubble
honor_labels: true
honor_timestamps: true
scrape_timeout: 60s
metrics_path: /metrics
scheme: http
follow_redirects: true
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_k8s_app, __meta_kubernetes_service_labelpresent_k8s_app]
separator: ;
regex: (hubble);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: hubble-metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: hubble-metrics
action: replace
- source_labels: [__address__]
separator: ;
regex: (.*)
modulus: 1
target_label: __tmp_hash
replacement: $1
action: hashmod
- source_labels: [__tmp_hash]
separator: ;
regex: "0"
replacement: $1
action: keep
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
follow_redirects: true
- job_name: integrations/cilium-enterprise/hubble-enterprise
honor_labels: true
honor_timestamps: true
scrape_timeout: 60s
metrics_path: /metrics
scheme: http
follow_redirects: true
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_instance, __meta_kubernetes_service_labelpresent_app_kubernetes_io_instance]
separator: ;
regex: (hubble-enterprise);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_managed_by,
__meta_kubernetes_service_labelpresent_app_kubernetes_io_managed_by]
separator: ;
regex: (Helm);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name, __meta_kubernetes_service_labelpresent_app_kubernetes_io_name]
separator: ;
regex: (hubble-enterprise);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_service_label_helm_sh_chart, __meta_kubernetes_service_labelpresent_helm_sh_chart]
separator: ;
regex: (hubble-enterprise-9999.9999.9999-dev);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: metrics
action: replace
- source_labels: [__meta_kubernetes_pod_node_name]
separator: ;
regex: (.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__address__]
separator: ;
regex: (.*)
modulus: 1
target_label: __tmp_hash
replacement: $1
action: hashmod
- source_labels: [__tmp_hash]
separator: ;
regex: "0"
replacement: $1
action: keep
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
follow_redirects: true
Full example configuration for Grafana Agent
Refer to the following Grafana Agent configuration for a complete example that contains all the snippets used for the Cilium Enterprise integration. This example also includes metrics that are sent to monitor your Grafana Agent instance.
integrations:
prometheus_remote_write:
- basic_auth:
password: <your_prom_pass>
username: <your_prom_user>
url: <your_prom_url>
agent:
enabled: true
relabel_configs:
- action: replace
source_labels:
- agent_hostname
target_label: instance
- action: replace
target_label: job
replacement: "integrations/agent-check"
metric_relabel_configs:
- action: keep
regex: (prometheus_target_sync_length_seconds_sum|prometheus_target_scrapes_.*|prometheus_target_interval.*|prometheus_sd_discovered_targets|agent_build.*|agent_wal_samples_appended_total|process_start_time_seconds)
source_labels:
- __name__
# Add here any snippet that belongs to the `integrations` section.
# For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.
logs:
configs:
- clients:
- basic_auth:
password: <your_loki_pass>
username: <your_loki_user>
url: <your_loki_url>
name: integrations
positions:
filename: /tmp/positions.yaml
scrape_configs:
# Add here any snippet that belongs to the `logs.configs.scrape_configs` section.
# For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.
metrics:
configs:
- name: integrations
remote_write:
- basic_auth:
password: <your_prom_pass>
username: <your_prom_user>
url: <your_prom_url>
scrape_configs:
# Add here any snippet that belongs to the `metrics.configs.scrape_configs` section.
# For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.
- job_name: integrations/cilium-enterprise/hubble-timescape-ingestor
honor_labels: true
honor_timestamps: true
scrape_timeout: 60s
metrics_path: /metrics
scheme: http
follow_redirects: true
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_instance, __meta_kubernetes_service_labelpresent_app_kubernetes_io_instance]
separator: ;
regex: (hubble-timescape);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name, __meta_kubernetes_service_labelpresent_app_kubernetes_io_name]
separator: ;
regex: (hubble-timescape-ingestor);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: metrics
action: replace
- source_labels: [__address__]
separator: ;
regex: (.*)
modulus: 1
target_label: __tmp_hash
replacement: $1
action: hashmod
- source_labels: [__tmp_hash]
separator: ;
regex: "0"
replacement: $1
action: keep
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
follow_redirects: true
- job_name: integrations/cilium-enterprise/hubble-timescape-server
honor_labels: true
honor_timestamps: true
scrape_timeout: 60s
metrics_path: /metrics
scheme: http
follow_redirects: true
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_instance, __meta_kubernetes_service_labelpresent_app_kubernetes_io_instance]
separator: ;
regex: (hubble-timescape);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name, __meta_kubernetes_service_labelpresent_app_kubernetes_io_name]
separator: ;
regex: (hubble-timescape-server);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: metrics
action: replace
- source_labels: [__address__]
separator: ;
regex: (.*)
modulus: 1
target_label: __tmp_hash
replacement: $1
action: hashmod
- source_labels: [__tmp_hash]
separator: ;
regex: "0"
replacement: $1
action: keep
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
follow_redirects: true
- job_name: integrations/cilium-enterprise/cilium-agent
honor_labels: true
honor_timestamps: true
scrape_timeout: 60s
metrics_path: /metrics
scheme: http
follow_redirects: true
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_k8s_app, __meta_kubernetes_service_labelpresent_k8s_app]
separator: ;
regex: (cilium);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_label_k8s_app]
separator: ;
regex: (.+)
target_label: k8s_app
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: metrics
action: replace
- source_labels: [__address__]
separator: ;
regex: (.*)
modulus: 1
target_label: __tmp_hash
replacement: $1
action: hashmod
- source_labels: [__tmp_hash]
separator: ;
regex: "0"
replacement: $1
action: keep
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
follow_redirects: true
- job_name: integrations/cilium-enterprise/cilium-operator
honor_labels: true
honor_timestamps: true
scrape_timeout: 60s
metrics_path: /metrics
scheme: http
follow_redirects: true
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_io_cilium_app, __meta_kubernetes_service_labelpresent_io_cilium_app]
separator: ;
regex: (operator);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_service_label_name, __meta_kubernetes_service_labelpresent_name]
separator: ;
regex: (cilium-operator);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_label_io_cilium_app]
separator: ;
regex: (.+)
target_label: io_cilium_app
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: metrics
action: replace
- source_labels: [__address__]
separator: ;
regex: (.*)
modulus: 1
target_label: __tmp_hash
replacement: $1
action: hashmod
- source_labels: [__tmp_hash]
separator: ;
regex: "0"
replacement: $1
action: keep
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
follow_redirects: true
- job_name: integrations/cilium-enterprise/hubble
honor_labels: true
honor_timestamps: true
scrape_timeout: 60s
metrics_path: /metrics
scheme: http
follow_redirects: true
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_k8s_app, __meta_kubernetes_service_labelpresent_k8s_app]
separator: ;
regex: (hubble);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: hubble-metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: hubble-metrics
action: replace
- source_labels: [__address__]
separator: ;
regex: (.*)
modulus: 1
target_label: __tmp_hash
replacement: $1
action: hashmod
- source_labels: [__tmp_hash]
separator: ;
regex: "0"
replacement: $1
action: keep
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
follow_redirects: true
- job_name: integrations/cilium-enterprise/hubble-enterprise
honor_labels: true
honor_timestamps: true
scrape_timeout: 60s
metrics_path: /metrics
scheme: http
follow_redirects: true
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_instance, __meta_kubernetes_service_labelpresent_app_kubernetes_io_instance]
separator: ;
regex: (hubble-enterprise);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_managed_by,
__meta_kubernetes_service_labelpresent_app_kubernetes_io_managed_by]
separator: ;
regex: (Helm);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name, __meta_kubernetes_service_labelpresent_app_kubernetes_io_name]
separator: ;
regex: (hubble-enterprise);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_service_label_helm_sh_chart, __meta_kubernetes_service_labelpresent_helm_sh_chart]
separator: ;
regex: (hubble-enterprise-9999.9999.9999-dev);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: metrics
action: replace
- source_labels: [__meta_kubernetes_pod_node_name]
separator: ;
regex: (.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__address__]
separator: ;
regex: (.*)
modulus: 1
target_label: __tmp_hash
replacement: $1
action: hashmod
- source_labels: [__tmp_hash]
separator: ;
regex: "0"
replacement: $1
action: keep
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
follow_redirects: true
global:
scrape_interval: 60s
wal_directory: /tmp/grafana-agent-wal
Dashboards
The Cilium Enterprise integration installs the following dashboards in your Grafana Cloud instance to help monitor your system.
- Cilium / Agent Overview
- Cilium / Components / API
- Cilium / Components / Agent
- Cilium / Components / BPF
- Cilium / Components / Conntrack
- Cilium / Components / Datapath
- Cilium / Components / External HA FQDN Proxy
- Cilium / Components / FQDN Proxy
- Cilium / Components / Identities
- Cilium / Components / Kubernetes
- Cilium / Components / L3 Policy
- Cilium / Components / L7 Proxy
- Cilium / Components / Network
- Cilium / Components / Nodes
- Cilium / Components / Policy
- Cilium / Components / Resource Utilization
- Cilium / Operator
- Cilium / Overview
- Hubble / Overview
- Hubble / Timescape
Cilium Overview
Cilium Overview (2)
Cilium Agent Overview
Alerts
The Cilium Enterprise integration includes the following useful alerts:
Cilium Endpoints
Alert | Description |
---|---|
CiliumAgentEndpointFailures | Warning: Cilium Agent endpoints in the invalid state. |
CiliumAgentEndpointUpdateFailure | Warning: API calls to Cilium Agent API to create or update Endpoints are failing. |
CiliumAgentContainerNetworkInterfaceApiErrorEndpointCreate | Info: Cilium Endpoint API endpoint rate limiter is reporting errors while doing endpoint create. |
CiliumAgentApiEndpointErrors | Warning: API calls to Cilium Endpoints API are failing due to server errors. |
Cilium IPAM
Alert | Description |
---|---|
CiliumOperatorExhaustedIpamIps | Critical: Cilium Operator has exhausted its IPAM IPs. |
CiliumOperatorLowAvailableIpamIps | Warning: Cilium Operator has used up over 90% of its available IPs. |
CiliumOperatorEniIpamErrors | Critical: Cilium Operator has high error rate while trying to create/attach ENIs for IPAM. |
Cilium Maps
Alert | Description |
---|---|
CiliumAgentMapOperationFailures | Warning: Cilium Agent is experiencing errors updating BPF maps on Agent Pod. |
CiliumAgentBpfMapPressure | Warning: Map on Cilium Agent Pod is currently experiencing high map pressure. |
Cilium NAT
Alert | Description |
---|---|
CiliumAgentNatTableFull | Critical: Cilium Agent Pod is dropping packets due to “No mapping for NAT masquerade” errors. |
Cilium API
Alert | Description |
---|---|
CiliumAgentApiHighErrorRate | Info: Cilium Agent API on Pod is experiencing a high error rate. |
Cilium Conntrack
Alert | Description |
---|---|
CiliumAgentConntrackTableFull | Critical: Ciliums conntrack map is failing on new insertions on Agent Pod. |
CiliumAgentConnTrackFailedGarbageCollectorRuns | Warning: Cilium Agent Conntrack GC runs are failing on Agent Pod. |
Cilium Drops
Alert | Description |
---|---|
CiliumAgentHighDeniedRate | Info: Cilium Agent is experiencing a high drop rate due to policy rule denies. |
Cilium Policy
Alert | Description |
---|---|
CiliumAgentPolicyMapPressure | Warning: Cilium Agent is experiencing high BPF map pressure. |
Cilium Identity
Alert | Description |
---|---|
CiliumNodeLocalHighIdentityAllocation | Warning: Cilium is using a very high percent (over 80%) of its maximum per-node identity limit (65535). |
RunningOutOfCiliumClusterIdentities | Warning: Cilium is using a very high percent of its maximum cluster identity limit (65280). |
Cilium Nodes
Alert | Description |
---|---|
CiliumUnreachableNodes | Info: Cilium Agent is reporting unreachable Nodes in the cluster. |
Metrics
The most important metrics provided by the Cilium Enterprise integration, which are used on the pre-built dashboards and Prometheus alerts, are as follows:
- cilium_agent_api_process_time_seconds_count
- cilium_agent_api_process_time_seconds_sum
- cilium_api_limiter_processed_requests_total
- cilium_bpf_map_ops_total
- cilium_bpf_map_pressure
- cilium_controllers_runs_duration_seconds_count
- cilium_controllers_runs_duration_seconds_sum
- cilium_controllers_runs_total
- cilium_datapath_conntrack_gc_duration_seconds_count
- cilium_datapath_conntrack_gc_duration_seconds_sum
- cilium_datapath_conntrack_gc_entries
- cilium_datapath_conntrack_gc_key_fallbacks_total
- cilium_datapath_conntrack_gc_runs_total
- cilium_drop_bytes_total
- cilium_drop_count_total
- cilium_endpoint_regeneration_time_stats_seconds_count
- cilium_endpoint_regeneration_time_stats_seconds_sum
- cilium_endpoint_regenerations_total
- cilium_endpoint_state
- cilium_errors_warnings_total
- cilium_forward_bytes_total
- cilium_forward_count_total
- cilium_identity
- cilium_ip_addresses
- cilium_k8s_client_api_calls_total
- cilium_k8s_client_api_latency_time_seconds_count
- cilium_k8s_client_api_latency_time_seconds_sum
- cilium_kubernetes_events_received_total
- cilium_kubernetes_events_total
- cilium_nodes_all_events_received_total
- cilium_nodes_all_num
- cilium_operator_ces_queueing_delay_seconds_bucket
- cilium_operator_ces_sync_errors_total
- cilium_operator_ec2_api_duration_seconds_bucket
- cilium_operator_identity_gc_entries
- cilium_operator_identity_gc_runs
- cilium_operator_ipam_allocation_ops
- cilium_operator_ipam_deficit_resolver_duration_seconds_bucket
- cilium_operator_ipam_interface_creation_ops
- cilium_operator_ipam_ips
- cilium_operator_ipam_k8s_sync_queued_total
- cilium_operator_ipam_nodes
- cilium_operator_ipam_resync_queued_total
- cilium_operator_ipam_resync_total
- cilium_operator_number_of_ceps_per_ces_sum
- cilium_operator_process_cpu_seconds_total
- cilium_operator_process_open_fds
- cilium_operator_process_resident_memory_bytes
- cilium_operator_process_virtual_memory_bytes
- cilium_policy
- cilium_policy_endpoint_enforcement_status
- cilium_policy_l7_denied_total
- cilium_policy_l7_forwarded_total
- cilium_policy_l7_received_total
- cilium_proxy_redirects
- cilium_proxy_upstream_reply_seconds_count
- cilium_proxy_upstream_reply_seconds_sum
- cilium_services_events_total
- cilium_triggers_policy_update_call_duration_seconds_count
- cilium_triggers_policy_update_call_duration_seconds_sum
- cilium_unreachable_nodes
- cilium_version
- hubble_dns_queries_total
- hubble_dns_response_types_total
- hubble_dns_responses_total
- hubble_drop_total
- hubble_flows_processed_total
- hubble_http_request_duration_seconds_bucket
- hubble_http_requests_total
- hubble_http_responses_total
- hubble_icmp_total
- hubble_port_distribution_total
- hubble_tcp_flags_total
- isovalent_external_dns_proxy_policy_l7_total
- isovalent_external_dns_proxy_processing_duration_seconds
- isovalent_external_dns_proxy_update_errors_total
- isovalent_external_dns_proxy_update_queue_size
- timescape_clickhouse_queries_duration_seconds_bucket
- timescape_clickhouse_queries_results_count
- timescape_clickhouse_queries_results_sum
- timescape_ingestor_flows_ingested_total
- timescape_ingestor_ingest_duration_seconds_bucket
- timescape_ingestor_ingest_running
- timescape_ingestor_ingestfilter_batch_duration_seconds_bucket
- timescape_ingestor_ingestfilter_filtered_errors_total
- timescape_ingestor_ingestfilter_filtered_skipped_total
- timescape_ingestor_ingestfilter_filtered_total
- timescape_ingestor_ingestlog_getinfo_queries
- up
Changelog
# 1.0.0 - June 2024
* Update Mixin to latest version
- Removed pod filter from alert rules
- Added thresholds for alerts using rate()
- Added aggregation label support
# 0.0.4 - November 2023
* Replaced Angular dashboard panels with React panels
# 0.0.3 - July 2023
* Added support for using the integration in the Grafana Cloud Kubernetes App
* Update all scrape intervals to be 60s
* Fix job name to correct value in static agent config
# 0.0.2 - January 2023
* Update mixin to latest version:
- Add new alert `CiliumOperatorEniIpamErrors` to alert on errors related to allocating new IPAM addresses and situations where nodes are experiencing IPAM exhaustion
- Fix alert conditions to trigger correctly
# 0.0.1 - October 2022
* Initial release
Cost
By connecting your Cilium Enterprise instance to Grafana Cloud, you might incur charges. To view information on the number of active series that your Grafana Cloud account uses for metrics included in each Cloud tier, see Active series and dpm usage and Cloud tier pricing.