We are facing and issue while trying to insert data to influxdb to be picked up from grafana. As of now there a test going on where we are inserting directly from collectd from a host to and influxdb db, the thing is that this is getting pretty huge and this is only information from 1 single host, we have hundreds of hosts in our PROD environment which is our main goal.
What we are trying to do is to insert only the data whenever we need to investigate an outage, for that we are planning to do it from the RAW files that collectd is creating right now. We came across a solution that is there but for NMON which does exactly what we need but for NMON files but I honestly don’t know anything about Golang which is the language they used for that, or at least I think so.
I found in this community that there is a PARSER plugin that is part of telegraf, but what I can’t find is how can I use it to take the information from a RAW file given by collectd and insert it into a influxdb.
I would really appreciate if someone can give me an advice on this matter.
Thanks in Advance.
OUTPUT SAMPLE:
################################################################################
# Collectl: V3.6.5-2 HiRes: 1 Options: -D
# Host: localhost DaemonOpts: -f /var/log/collectl -r00:00,7,60 -m -F60 -s+YZ -i60
# Booted: 1548722801.34 [20190128-18:46:41]
# Distro: Red Hat Enterprise Linux Server release 6.9 (Santiago) Platform: VMware Virtual Platform
# Date: 20190130-230000 Secs: 1548910800 TZ: -0600
# SubSys: bcdfijmnstYZ Options: Interval: 60:60 NumCPUs: 10 NumBud: 3 Flags: ix
# Filters: NfsFilt: EnvFilt: TcpFilt: ituc
# HZ: 100 Arch: x86_64-linux-thread-multi PageSize: 4096
# Cpu: GenuineIntel Speed(MHz): 2992.968 Cores: 1 Siblings: 1 Nodes: 1
# Kernel: 2.6.32-696.20.1.el6.x86_64 Memory: 198339428 kB Swap: 17825788 kB
# NumDisks: 22 DiskNames: sda sdb sdc sde sdd dm-0 dm-1 dm-2 dm-3 dm-4 dm-5 dm-6 dm-7 dm-8 dm-9 dm-10 dm-11 dm-12 dm-13 dm-14 dm-15 dm-16
# NumNets: 3 NetNames: lo:?? eth0:10000 eth1:10000
# NumSlabs: 201 Version: 2.1
# SCSI: CD:1:00:00:00 DA:2:00:00:00 DA:2:00:01:00 DA:2:00:02:00 DA:2:00:03:00 DA:2:00:04:00
################################################################################
>>> 1548910800.001 <<<
buddy Node 0, zone DMA 1 1 1 3 3 2 1 1 0 0 2
buddy Node 0, zone DMA32 14 18 15 17 14 7 7 11 8 4 106
buddy Node 0, zone Normal 403 386 5583 3572 13604 10598 7080 3627 2055 2 27
cpu 22087348 256 7084173 153333789 15626 4857 2989099 0 0
cpu0 2343703 23 756440 15044112 1503 394 389489 0 0
cpu1 1424427 40 403584 16804532 1978 2 13682 0 0
cpu2 1400317 30 399191 16835804 1824 1 12608 0 0
cpu3 2373492 20 777245 15024216 1405 441 357515 0 0
cpu4 2305879 21 754504 15129573 1544 395 334811 0 0
cpu5 2406073 20 826459 14953667 1377 1854 337925 0 0
cpu6 2524873 19 813584 14794741 1330 456 388679 0 0
cpu7 2377199 26 774277 14997185 1586 440 369269 0 0
cpu8 2431979 19 818481 14856251 1454 501 415840 0 0
cpu9 2499402 33 760403 14893703 1620 370 369277 0 0
intr 5904658682
ctxt 4292174746
processes 1824586
procs_running 3
procs_blocked 0
int 0: 844 0 0 0 0 0 0 0 0 0 IO-APIC-edge timer
int 1: 10 1146 0 0 0 0 0 0 0 0 IO-APIC-edge i8042
int 8: 1 0 0 0 0 0 0 0 0 0 IO-APIC-edge rtc0
int 9: 0 0 0 0 0 0 0 0 0 0 IO-APIC-fasteoi acpi
int 12: 110 0 0 1080 0 0 0 0 0 0 IO-APIC-edge i8042
int 14: 0 0 0 0 0 0 0 0 0 0 IO-APIC-edge ata_piix
int 15: 95 0 0 0 0 639862 0 0 0 0 IO-APIC-edge ata_piix
int 16: 4 0 0 0 0 0 0 0 0 2 IO-APIC-fasteoi vmwgfx
int 24: 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge pciehp
int 25: 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge pciehp
int 26: 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge pciehp
int 27: 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge pciehp
int 28: 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge pciehp
int 29: 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge pciehp
int 30: 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge pciehp
int 31: 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge pciehp
int 32: 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge pciehp
int 33: 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge pciehp
int 34: 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge pciehp
int 35: 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge pciehp
int 36: 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge pciehp
int 37: 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge pciehp
int 38: 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge pciehp
int 39: 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge pciehp
int 40: 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge pciehp
disk 253 3 dm-3 817 0 6530 1019 1 0 8 6 0 187 1025
disk 253 4 dm-4 261 0 3298 154 78648 0 629184 94613 0 12780 94788
disk 253 5 dm-5 122 0 970 30 1 0 8 0 0 30 30
disk 253 6 dm-6 128 0 1018 49 1 0 8 6 0 55 55
disk 253 7 dm-7 122 0 970 43 1 0 8 0 0 43 43
disk 253 8 dm-8 14705 0 173314 18512 30827989 0 246623912 309099632 0 580567 309838167
disk 253 9 dm-9 257 0 2050 167 1 0 8 0 0 80 167
disk 253 10 dm-10 142 0 1130 58 112 0 896 81 0 107 139
disk 253 11 dm-11 26758 0 1183658 26638 9156 0 73248 27014 0 15012 53652
disk 253 12 dm-12 133 0 1058 59 6 0 48 2 0 49 61
disk 253 13 dm-13 133 0 1058 56 6 0 48 2 0 46 58
disk 253 14 dm-14 5088 0 615178 3932 107319 0 858552 1285013 0 3111 1288946
disk 253 15 dm-15 15240 0 410298 10749 1538387 0 12307096 7551698 0 121712 7567479
disk 253 16 dm-16 544 0 20778 753 989864 0 7918912 2153755 0 479168 2155269
load 0.21 0.16 0.10 1/2757 24089
tcp-Ip: Forwarding DefaultTTL InReceives InHdrErrors InAddrErrors ForwDatagrams InUnknownProtos InDiscards InDelivers OutRequests OutDiscards OutNoRoutes ReasmTimeout ReasmReqds ReasmOKs ReasmFails FragOKs FragFails FragCreates
tcp-Ip: 2 64 2345808100 0 65987 0 0 0 2345023989 2542097859 0 88 0 1374787 657103 0 658849 0 1378045
tcp-Icmp: InMsgs InErrors InDestUnreachs InTimeExcds InParmProbs InSrcQuenchs InRedirects InEchos InEchoReps InTimestamps InTimestampReps InAddrMasks InAddrMaskReps OutMsgs OutErrors OutDestUnreachs OutTimeExcds OutParmProbs OutSrcQuenchs OutRedirects OutEchos OutEchoReps OutTimestamps OutTimestampReps OutAddrMasks OutAddrMaskReps
tcp-Icmp: 9644 24 1453 0 0 0 0 4223 3966 2 0 0 0 10125 0 1934 0 0 0 0 3966 4223 0 2 0 0
tcp-Tcp: RtoAlgorithm RtoMin RtoMax MaxConn ActiveOpens PassiveOpens AttemptFails EstabResets CurrEstab InSegs OutSegs RetransSegs InErrs OutRsts
tcp-Tcp: 1 200 120000 -1 189518 203747 116 1705 1538 2337716464 2532808789 6750 1 1108
tcp-Udp: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors
tcp-Udp: 7155937 2287 12361 9272237 1 0
fs-ds 279615 266755 45 0 0 0
fs-is 121177 118
fs-fnr 9728 0 6815744
nfsc-net 0 0 0 0
nfsc-rpc 499842 1085 499886
nfsc-proc2 18 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
nfsc-proc3 22 0 105381 0 2333 383140 6 1704 0 0 0 0 0 0 0 0 0 0 44 7201 20 10 0
Telegraf's docs
show an example of how to read a collectd file,
you use the file input and use data_format = "collectd" to use the collectd parser.
Side note, InfluxDB can accept CollectD directly. (this may require this file which didn't seem to be included in my installation)
For rate limiting, check out collectd's interval, or if you prefer using telegraf, it also has such an option.
I'm unsure about influxdb's options for this, but their downsampling and retention features may be of interest to you.