Metriken empfangen und anzeigen
Der im Abschnitt cc-metric-collector einrichten konfigurierte Collector sendet zwar bereits Werte, allerdings akzeptiert cc-backend sie nur, wenn die Metriken explizit hinterlegt sind. In diesem Abschnitt wird beschrieben, wie neue oder geänderte Metriken auf dem Monitoring-Server hinterlegt werden, damit
- der in
cc-backendkonfigurierte Metric-Store die Daten entgegennimmt und - das Webinterface (
cc-backend) die Metriken über Einträge in dercluster.jsonerkennt.
Kurzfassung: Jede neue Metrik benötigt einen Eintrag in der
metricConfigdercluster.json. Die Metric-Store-Laufzeitparameter stehen incc-backend/config.jsonuntermetric-store.
1. Metric-Store im cc-backend prüfen
Dateipfad (Standardinstallation gemäß Guide):
$INSTALL_DIR/cc-backend/config.json
Der Metric-Store wird über den Abschnitt metric-store in cc-backend/config.json konfiguriert:
"metric-store": {
"checkpoints": {
"file-format": "json",
"directory": "./var/checkpoints"
},
"memory-cap": 100,
"retention-in-memory": "48h",
"cleanup": {
"mode": "archive",
"directory": "./var/archive"
}
}
Felder: - retention-in-memory: Zeitraum, in dem Messdaten im RAM gehalten werden. - checkpoints: Ablage und Format der Sicherungspunkte. - cleanup: Verhalten für alte Daten. Für 1.5.3 ist archive bzw. die stabile Retention-Policy der Installation zu prüfen; bei Problemen empfiehlt upstream weiterhin eine konservative Delete-Policy.
Nach Änderungen muss cc-backend neu gestartet werden:
systemctl restart clustercockpit.service
2. cc-backend: cluster.json aktualisieren
Dateipfad:
$INSTALL_DIR/cc-backend/var/job-archive/$CLUSTER_NAME/cluster.json
cluster.json beschreibt sowohl die Subcluster als auch alle Metriken, die im Webinterface angezeigt werden sollen. Jede Metrik steht als Objekt innerhalb des Arrays metricConfig.
Minimaler Eintrag
{
"name": "cpu_load",
"unit": { "base": "" },
"scope": "node",
"aggregation": "avg",
"timestep": 60,
"peak": 48,
"normal": 48,
"caution": 10,
"alert": 1
}
Wichtige Felder: - name: Muss exakt mit dem Namen übereinstimmen, der nach dem Routing (z. B. rename_messages) publiziert wird. - unit: Basiseinheit (optional prefix: "M", "G", ...), konsistent zu den in router.json gesetzten Einheiten. - scope: Granularität (node, socket, hwthread, memoryDomain, accelerator). - aggregation: Üblicherweise avg für Zustände oder sum für kumulative Metriken. - timestep: Anzeigeintervall (in Sekunden), sollte zum interval des cc-metric-collector passen. - peak, normal, caution, alert: Schwellenwerte für das UI. Graphen zwischen normal und caution bleiben neutral. Werte zwischen caution und alert werden gelb markiert (= genauer hinschauen), Werte unter alert rot (= sofort reagieren).
Konfigurationsmöglichkeiten
Alert-Logik umkehren
Durch Setzen von "lowerIsBetter": true wird der Alarm nicht bei zu kleinen, sondern bei zu hohen Werten ausgelöst. Das ist z. B. für cpu_load_core, Netzwerkbandbreiten oder IOPS sinnvoll.
Metrik nur für bestimmte Subcluster
Um z. B. GPU-Metriken nur für Subcluster anzuzeigen, die auch GPUs verbaut haben, können Metriken aus einzelnen Subclustern entfernt werden:
"name": "nv_compute_processes",
"unit": {
"base": "processes"
},
"scope": "accelerator",
"aggregation": "sum",
"timestep": 60,
"peak": 100,
"normal": 50,
"caution": 80,
"alert": 90,
"subClusters": [
{
"name": "cpu",
"remove": true
},
{
"name": "cpu_abs2",
"remove": true
},
{
"name": "fatcpu",
"remove": true
}
]
},
Anmerkung: Gelöschte Metriken sind in der Jobview nicht mehr sichtbar und können bei der Metrikauswahl auch nicht ausgewählt werden. In der Node-View für Administratoren tauchen blaue Felder auf, die darauf hinweisen, dass die Metrik für diesen Subcluster deaktiviert ist:
Unterschiedliche Thresholds für verschiedene Subcluster
Bei unterschiedlicher Ausstattung mit Cores, Memory, GPU-Memory usw. können die Thresholds pro Subcluster angegeben werden. Alle nicht genannten Subcluster erhalten die Konfiguration aus dem Hauptteil:
{
"name": "cpu_load",
"unit": {
"base": "load"
},
"scope": "node",
"aggregation": "avg",
"timestep": 60,
"peak": 48,
"normal": 48,
"caution": 10,
"alert": 1,
"subClusters": [
{
"name": "fatcpu",
"peak": 96,
"normal": 96,
"caution": 10,
"alert": 1
},
{
"name": "fatgpu",
"peak": 96,
"normal": 96,
"caution": 10,
"alert": 1
}
]
},
Footprint-Metriken definieren
Um die Metriken zu definieren, die im Footprint und im Polar Plot angezeigt werden, wird "footprint": "avg" (oder sum, je nach aggregation) ergänzt.
Darstellung im Footprint:
Darstellung im Polar Plot:
Hat die Metrik den Eintrag "lowerIsBetter": true, wird dies durch einen Pfeil nach links dargestellt. In diesem Beispiel wurden ausschließlich Metriken verwendet, deren Scope hwthread ist, da nur diese Metriken bei shared Nodes aussagekräftig sind.
Energy Footprint definieren
Um festzulegen, ob eine Metrik in den Energy Footprint einfließt, wird "energy": "power" ergänzt, wenn die Einheit Watt ist, oder "energy": "energy", wenn die Einheit Joule ist. Aus allen energy-Metriken wird ein Gesamtverbrauch errechnet.
Für CPU-only-Jobs:
Wenn GPUs zum Einsatz kommen, wird deren Verbrauch einzeln aufgeführt:
Auf Intel-Systemen ist es ebenfalls möglich, den Stromverbrauch des Arbeitsspeichers auszulesen, bei AMD stehen keine Counter dafür zur Verfügung.
Optional kann in der config.json von cc-backend der Parameter emission-constant gesetzt werden. Er beschreibt den CO₂-Faktor des Rechenzentrumsstroms in g/kWh. Auf Basis dieses Werts und des gemessenen Energieverbrauchs wird automatisch die (theoretische) CO₂-Emission pro Job berechnet und im UI angezeigt.
Änderungen aktivieren
Damit die Änderungen wirksam werden, muss cc-backend neu gestartet werden:
systemctl restart clustercockpit.service
3. Prüfen, ob Metriken ankommen
Für jeden Host, auf dem cc-metric-collector läuft, sollten sich nach und nach die Graphen mit Daten füllen. Falls stattdessen ein gelbes Feld angezeigt wird, kommen keine Daten an:
Es sollte überprüft werden, ob die Metrik vom cc-metric-collector mit diesem Namen gesendet und von cc-backend über /api/write/ empfangen wird.
Beispielkonfigurationen
Die folgenden JSON-Dateien zeigen eine produktive Beispielkonfiguration für einen Cluster mit CPU-, GPU-, Speicher-, Netzwerk- und Dateisystemmetriken.
cc-backend/config.json
Inhalt anzeigen
{
"main": {
"addr": "127.0.0.1:8443",
"api-allowed-ips": ["*"],
"short-running-jobs-duration": 30,
"enable-job-taggers": false,
"nodestate-retention": {
"policy": "move",
"target-kind": "file",
"target-path": "./var/nodestate-archive"
},
"resampling": {
"minimum-points": 600,
"trigger": 180,
"resolutions": [240, 60]
}
},
"auth": {
"ldap": {
"url": "ldaps://openldap.ruhr-uni-bochum.de:636",
"user-base": "ou=users,dc=ruhr-uni-bochum,dc=de",
"search-dn": "cn=hpc-web,ou=sicherheit,dc=ruhr-uni-bochum,dc=de",
"user-bind": "uid={username},ou=users,dc=ruhr-uni-bochum,dc=de",
"user-filter": "(&(objectclass=posixAccount)(memberOf=cn=elysium,ou=groups,dc=ruhr-uni-bochum,dc=de))",
"syncUserOnLogin": true
},
"jwts": {
"max-age": ""
}
},
"cron": {
"commit-job-worker": "1m",
"duration-worker": "5m",
"footprint-worker": "10m"
},
"archive": {
"kind": "file",
"path": "/opt/monitoring/job-archive"
},
"metric-store": {
"checkpoints": {
"file-format": "json",
"directory": "/opt/monitoring/cc-backend/var/checkpoints"
},
"memory-cap": 100,
"retention-in-memory": "48h",
"cleanup": {
"mode": "archive",
"directory": "/opt/monitoring/cc-backend/var/archive"
}
},
"ui-file": "ui-config.json"
}
job-archive/cluster.json
Inhalt anzeigen
{
"name": "elysium",
"metricConfig": [
{
"name": "cpu_load",
"unit": {
"base": "load"
},
"scope": "node",
"aggregation": "avg",
"timestep": 60,
"peak": 48,
"normal": 48,
"caution": 10,
"alert": 1,
"subClusters": [
{ "name": "fatcpu", "peak": 96, "normal": 96, "caution": 10, "alert": 1 },
{ "name": "fatgpu", "peak": 96, "normal": 96, "caution": 10, "alert": 1 },
{ "name": "h200gpu", "peak": 128, "normal": 128, "caution": 10, "alert": 1 },
{ "name": "tcpu", "peak": 192, "normal": 192, "caution": 10, "alert": 1 },
{ "name": "tl40sgpu", "peak": 64, "normal": 64, "caution": 10, "alert": 1 },
{ "name": "th200gpu", "peak": 64, "normal": 64, "caution": 10, "alert": 1 },
{ "name": "tvis", "peak": 128, "normal": 128, "caution": 10, "alert": 1 }
]
},
{
"name": "cpu_user",
"unit": {
"base": "%",
"description": "Percentage of CPU time spent in user mode"
},
"scope": "hwthread",
"aggregation": "avg",
"footprint": "avg",
"timestep": 60,
"peak": 100,
"normal": 70,
"caution": 50,
"alert": 30
},
{
"name": "cpu_load_core",
"unit": {
"base": "load"
},
"scope": "hwthread",
"aggregation": "avg",
"timestep": 60,
"peak": 12,
"normal": 1.1,
"caution": 2,
"alert": 3,
"lowerIsBetter": true
},
{
"name": "mem_used",
"unit": {
"base": "B",
"prefix": "G"
},
"scope": "memoryDomain",
"aggregation": "sum",
"timestep": 60,
"peak": 384,
"normal": 384,
"caution": 384,
"alert": 384,
"subClusters": [
{ "name": "fatcpu", "peak": 2600, "normal": 2304, "caution": 2304, "alert": 2304 },
{ "name": "cpu_abs2", "peak": 900, "normal": 768, "caution": 768, "alert": 768 },
{ "name": "fatgpu", "peak": 1500, "normal": 1152, "caution": 1152, "alert": 1152 },
{ "name": "vis", "peak": 1500, "normal": 1152, "caution": 1152, "alert": 1152 },
{ "name": "h200gpu", "peak": 2700, "normal": 2304, "caution": 2304, "alert": 2304 },
{ "name": "tcpu", "peak": 1800, "normal": 1536, "caution": 1536, "alert": 1536 },
{ "name": "tl40sgpu", "peak": 900, "normal": 768, "caution": 768, "alert": 768 },
{ "name": "th200gpu", "peak": 1800, "normal": 1536, "caution": 1536, "alert": 1536 },
{ "name": "tvis", "peak": 900, "normal": 768, "caution": 768, "alert": 768 }
]
},
{
"name": "mem_bw",
"unit": {
"base": "B/s",
"prefix": "G"
},
"scope": "socket",
"aggregation": "sum",
"timestep": 60,
"peak": 700,
"normal": 591,
"caution": 591,
"alert": 591,
"subClusters": [
{ "name": "fatcpu", "peak": 900, "normal": 778, "caution": 778, "alert": 778 },
{ "name": "fatgpu", "peak": 900, "normal": 779, "caution": 779, "alert": 779 },
{ "name": "h200gpu", "peak": 900, "normal": 790, "caution": 790, "alert": 790 },
{ "name": "tcpu", "peak": 1000, "normal": 892, "caution": 892, "alert": 892 },
{ "name": "th200gpu", "peak": 1000, "normal": 864, "caution": 864, "alert": 864 },
{ "name": "tvis", "peak": 1300, "normal": 1148, "caution": 1148, "alert": 1148 }
]
},
{
"name": "net_bytes_in",
"unit": {
"base": "B/s",
"prefix": "M"
},
"scope": "node",
"aggregation": "sum",
"timestep": 60,
"lowerIsBetter": true,
"peak": 125,
"normal": 10,
"caution": 20,
"alert": 40
},
{
"name": "net_bytes_out",
"unit": {
"base": "B/s",
"prefix": "M"
},
"scope": "node",
"aggregation": "sum",
"lowerIsBetter": true,
"timestep": 60,
"peak": 125,
"normal": 10,
"caution": 20,
"alert": 40
},
{
"name": "net_pkts_in",
"unit": {
"base": "packets/s"
},
"scope": "node",
"aggregation": "sum",
"timestep": 60,
"peak": 100,
"normal": 100,
"caution": 100,
"alert": 100
},
{
"name": "net_pkts_out",
"unit": {
"base": "packets/s"
},
"scope": "node",
"aggregation": "sum",
"timestep": 60,
"peak": 100,
"normal": 100,
"caution": 100,
"alert": 100
},
{
"name": "disk_free",
"unit": { "base": "B", "prefix": "G" },
"scope": "node",
"aggregation": "sum",
"timestep": 60,
"peak": 3755,
"normal": 1878,
"caution": 939,
"alert": 376,
"subClusters": [
{ "name": "cpu", "peak": 876, "normal": 876, "caution": 219, "alert": 88 },
{ "name": "cpu_abs2", "peak": 3755, "normal": 3755, "caution": 939, "alert": 376 },
{ "name": "gpu", "peak": 1836, "normal": 1836, "caution": 459, "alert": 184 },
{ "name": "fatcpu", "peak": 1836, "normal": 1836, "caution": 459, "alert": 184 },
{ "name": "fatgpu", "peak": 1836, "normal": 1836, "caution": 459, "alert": 184 },
{ "name": "vis", "peak": 876, "normal": 876, "caution": 219, "alert": 88 }
]
},
{
"name": "io_reads",
"unit": {
"base": "ops/s"
},
"scope": "node",
"aggregation": "sum",
"timestep": 60,
"peak": 100,
"normal": 100,
"caution": 100,
"alert": 100
},
{
"name": "io_writes",
"unit": {
"base": "ops/s"
},
"scope": "node",
"aggregation": "sum",
"timestep": 60,
"peak": 100,
"normal": 100,
"caution": 100,
"alert": 100
},
{
"name": "lustre_open",
"unit": {
"base": "operations"
},
"scope": "node",
"aggregation": "sum",
"timestep": 60,
"peak": 50000,
"normal": 25000,
"caution": 40000,
"alert": 45000
},
{
"name": "lustre_close",
"unit": {
"base": "operations"
},
"scope": "node",
"aggregation": "sum",
"timestep": 60,
"peak": 50000,
"normal": 25000,
"caution": 40000,
"alert": 45000
},
{
"name": "lustre_statfs",
"unit": {
"base": "operations"
},
"scope": "node",
"aggregation": "sum",
"timestep": 60,
"peak": 100,
"normal": 50,
"caution": 80,
"alert": 90
},
{
"name": "lustre_read_bw",
"unit": {
"base": "B/s",
"prefix": "M"
},
"scope": "node",
"aggregation": "sum",
"timestep": 60,
"peak": 60000,
"normal": 60000,
"caution": 60000,
"alert": 60000
},
{
"name": "lustre_write_bw",
"unit": {
"base": "B/s",
"prefix": "M"
},
"scope": "node",
"aggregation": "sum",
"timestep": 60,
"peak": 60000,
"normal": 60000,
"caution": 60000,
"alert": 60000
},
{
"name": "nfs4_open",
"unit": {
"base": "operations"
},
"scope": "node",
"aggregation": "sum",
"timestep": 60,
"peak": 100,
"normal": 100,
"caution": 100,
"alert": 100
},
{
"name": "nfs4_close",
"unit": {
"base": "operations"
},
"scope": "node",
"aggregation": "sum",
"timestep": 60,
"peak": 100,
"normal": 100,
"caution": 100,
"alert": 100
},
{
"name": "nfsio_nread",
"unit": {
"base": "B/s",
"prefix": "M"
},
"scope": "node",
"aggregation": "sum",
"timestep": 60,
"peak": 100,
"normal": 100,
"caution": 100,
"alert": 100
},
{
"name": "nfsio_nwrite",
"unit": {
"base": "B/s",
"prefix": "M"
},
"scope": "node",
"aggregation": "sum",
"timestep": 60,
"peak": 100,
"normal": 100,
"caution": 100,
"alert": 100
},
{
"name": "numastats_interleave_hit",
"unit": {
"base": "count/s"
},
"scope": "memoryDomain",
"aggregation": "sum",
"timestep": 60,
"peak": 100,
"normal": 100,
"caution": 100,
"alert": 100
},
{
"name": "numastats_local_node",
"unit": {
"base": "count/s"
},
"scope": "memoryDomain",
"aggregation": "sum",
"timestep": 60,
"peak": 100,
"normal": 100,
"caution": 100,
"alert": 100
},
{
"name": "numastats_numa_foreign",
"unit": {
"base": "count/s"
},
"scope": "memoryDomain",
"aggregation": "sum",
"timestep": 60,
"peak": 100,
"normal": 100,
"caution": 100,
"alert": 100
},
{
"name": "numastats_numa_hit",
"unit": {
"base": "count/s"
},
"scope": "memoryDomain",
"aggregation": "sum",
"timestep": 60,
"peak": 100,
"normal": 100,
"caution": 100,
"alert": 100
},
{
"name": "numastats_numa_miss",
"unit": {
"base": "count/s"
},
"scope": "memoryDomain",
"aggregation": "sum",
"timestep": 60,
"peak": 100,
"normal": 100,
"caution": 100,
"alert": 100
},
{
"name": "numastats_other_node",
"unit": {
"base": "count/s"
},
"scope": "memoryDomain",
"aggregation": "sum",
"timestep": 60,
"peak": 100,
"normal": 100,
"caution": 100,
"alert": 100
},
{
"name": "ib_recv",
"unit": {
"base": "B/s",
"prefix": "M"
},
"scope": "node",
"aggregation": "sum",
"timestep": 60,
"peak": 12000,
"normal": 10000,
"caution": 0,
"alert": 0
},
{
"name": "ib_xmit",
"unit": {
"base": "B/s",
"prefix": "M"
},
"scope": "node",
"aggregation": "sum",
"timestep": 60,
"peak": 12000,
"normal": 10000,
"caution": 0,
"alert": 0
},
{
"name": "ib_recv_pkts",
"unit": {
"base": "packets/s"
},
"scope": "node",
"aggregation": "sum",
"timestep": 60,
"peak": 1100000,
"normal": 800000,
"caution": 0,
"alert": 0
},
{
"name": "ib_xmit_pkts",
"unit": {
"base": "packets/s"
},
"scope": "node",
"aggregation": "sum",
"timestep": 60,
"peak": 1100000,
"normal": 800000,
"caution": 0,
"alert": 0
},
{
"name": "nv_compute_processes",
"unit": {
"base": "processes"
},
"scope": "accelerator",
"aggregation": "sum",
"timestep": 60,
"peak": 100,
"normal": 50,
"caution": 80,
"alert": 90,
"subClusters": [
{ "name": "cpu", "remove": true },
{ "name": "cpu_abs2", "remove": true },
{ "name": "fatcpu", "remove": true },
{ "name": "tcpu", "remove": true }
]
},
{
"name": "acc_utilization",
"unit": {
"base": "%",
"description": "GPU Utilization Percentage"
},
"scope": "accelerator",
"aggregation": "avg",
"footprint": "avg",
"timestep": 60,
"peak": 100,
"normal": 90,
"caution": 70,
"alert": 60,
"subClusters": [
{ "name": "cpu", "remove": true },
{ "name": "cpu_abs2", "remove": true },
{ "name": "fatcpu", "remove": true },
{ "name": "tcpu", "remove": true }
]
},
{
"name": "acc_mem_used",
"unit": {
"base": "B",
"prefix": "G"
},
"scope": "accelerator",
"aggregation": "sum",
"timestep": 60,
"peak": 80,
"normal": 24,
"caution": 38,
"alert": 45,
"subClusters": [
{ "name": "cpu", "remove": true },
{ "name": "cpu_abs2", "remove": true },
{ "name": "fatcpu", "remove": true },
{ "name": "tcpu", "remove": true },
{ "name": "gpu", "scope": "accelerator", "aggregation": "sum", "timestep": 60, "peak": 80, "normal": 70, "caution": 71, "alert": 72 },
{ "name": "fatgpu", "scope": "accelerator", "aggregation": "sum", "timestep": 60, "peak": 840, "normal": 638, "caution": 639, "alert": 640 },
{ "name": "vis", "scope": "accelerator", "aggregation": "sum", "timestep": 60, "peak": 48, "normal": 47, "caution": 47, "alert": 48 },
{ "name": "h200gpu", "scope": "accelerator", "aggregation": "sum", "timestep": 60, "peak": 1512, "normal": 1150, "caution": 1151, "alert": 1152 }
]
},
{
"name": "acc_power",
"unit": {
"base": "W"
},
"scope": "accelerator",
"aggregation": "sum",
"timestep": 60,
"energy": "power",
"peak": 300,
"normal": 150,
"caution": 250,
"alert": 280,
"subClusters": [
{ "name": "cpu", "remove": true },
{ "name": "cpu_abs2", "remove": true },
{ "name": "fatcpu", "remove": true },
{ "name": "tcpu", "remove": true },
{ "name": "gpu", "scope": "accelerator", "aggregation": "sum", "energy": "power", "timestep": 60, "peak": 600, "normal": 493, "caution": 494, "alert": 495 },
{ "name": "fatgpu", "scope": "accelerator", "aggregation": "sum", "energy": "power", "timestep": 60, "peak": 6000, "normal": 5580, "caution": 5590, "alert": 5600 },
{ "name": "vis", "scope": "accelerator", "aggregation": "sum", "energy": "power", "timestep": 60, "peak": 400, "normal": 290, "caution": 295, "alert": 300 },
{ "name": "h200gpu", "scope": "accelerator", "aggregation": "sum", "energy": "power", "timestep": 60, "peak": 6000, "normal": 5580, "caution": 5590, "alert": 5600 }
]
},
{
"name": "acc_mem_util",
"unit": {
"base": "%",
"description": "GPU Memory Utilization Percentage"
},
"scope": "accelerator",
"aggregation": "avg",
"timestep": 60,
"peak": 100,
"normal": 80,
"caution": 50,
"alert": 20,
"subClusters": [
{ "name": "cpu", "remove": true },
{ "name": "cpu_abs2", "remove": true },
{ "name": "fatcpu", "remove": true },
{ "name": "tcpu", "remove": true }
]
},
{
"name": "clock",
"unit": {
"base": "MHz"
},
"scope": "hwthread",
"aggregation": "avg",
"timestep": 60,
"peak": 4980,
"normal": 4150,
"caution": 4150,
"alert": 4150,
"subClusters": [
{ "name": "fatcpu", "peak": 4560, "normal": 3800, "caution": 3800, "alert": 3800 },
{ "name": "fatgpu", "peak": 4560, "normal": 3800, "caution": 3800, "alert": 3800 },
{ "name": "h200gpu", "peak": 4500, "normal": 3750, "caution": 3750, "alert": 3750 },
{ "name": "tcpu", "peak": 4440, "normal": 3700, "caution": 3700, "alert": 3700 },
{ "name": "th200gpu", "peak": 5280, "normal": 4400, "caution": 4400, "alert": 4400 },
{ "name": "tl40sgpu", "peak": 5160, "normal": 4300, "caution": 4300, "alert": 4300 },
{ "name": "tvis", "peak": 5160, "normal": 4300, "caution": 4300, "alert": 4300 }
]
},
{
"name": "ipc",
"unit": {
"base": "IPC"
},
"scope": "hwthread",
"aggregation": "avg",
"footprint": "avg",
"timestep": 60,
"peak": 6,
"normal": 6,
"caution": 1,
"alert": 0
},
{
"name": "core_power",
"unit": {
"base": "W"
},
"scope": "hwthread",
"aggregation": "sum",
"energy": "power",
"timestep": 60,
"peak": 480,
"normal": 400,
"caution": 400,
"alert": 400,
"subClusters": [
{ "name": "fatcpu", "peak": 600, "normal": 580, "caution": 580, "alert": 580 },
{ "name": "fatgpu", "peak": 600, "normal": 580, "caution": 580, "alert": 580 },
{ "name": "h200gpu", "peak": 800, "normal": 720, "caution": 720, "alert": 720 },
{ "name": "tcpu", "peak": 800, "normal": 640, "caution": 640, "alert": 640 },
{ "name": "th200gpu", "peak": 480, "normal": 420, "caution": 420, "alert": 420 },
{ "name": "tl40sgpu", "peak": 300, "normal": 300, "caution": 300, "alert": 300 },
{ "name": "tvis", "peak": 600, "normal": 600, "caution": 600, "alert": 600 }
]
},
{
"name": "flops_any",
"unit": {
"base": "FLOPS",
"prefix": "G"
},
"scope": "hwthread",
"aggregation": "sum",
"footprint": "avg",
"timestep": 60,
"peak": 3200,
"normal": 3200,
"caution": 107,
"alert": 54,
"subClusters": [
{ "name": "fatcpu", "peak": 5800, "normal": 5809, "caution": 194, "alert": 97 },
{ "name": "fatgpu", "peak": 5800, "normal": 5796, "caution": 194, "alert": 97 },
{ "name": "h200gpu", "peak": 7600, "normal": 7607, "caution": 254, "alert": 127 },
{ "name": "tcpu", "peak": 22400, "normal": 22393, "caution": 747, "alert": 374 },
{ "name": "th200gpu", "peak": 9000, "normal": 8919, "caution": 298, "alert": 149 },
{ "name": "tl40sgpu", "peak": 7900, "normal": 7860, "caution": 262, "alert": 131 },
{ "name": "tvis", "peak": 16000, "normal": 15758, "caution": 526, "alert": 263 }
]
},
{
"name": "flops_bf16",
"unit": {
"base": "FLOPS",
"prefix": "G"
},
"scope": "hwthread",
"aggregation": "sum",
"footprint": "avg",
"timestep": 60,
"peak": 3000,
"normal": 3000,
"caution": 100,
"alert": 50,
"subClusters": [
{"name": "cpu", "remove": true },
{ "name": "cpu_abs2", "remove": true },
{ "name": "fatcpu", "remove": true },
{ "name": "gpu", "remove": true },
{ "name": "fatgpu", "remove": true },
{ "name": "vis", "remove": true },
{ "name": "h200gpu", "remove": true },
{ "name": "tcpu", "peak": 53800, "normal": 44786, "caution": 1493, "alert": 746 },
{ "name": "tl40sgpu", "peak": 18900, "normal": 15720, "caution": 524, "alert": 262 },
{ "name": "th200gpu", "peak": 21500, "normal": 17838, "caution": 595, "alert": 297 },
{ "name": "tvis", "peak": 37900, "normal": 31516, "caution": 1051, "alert": 525 }
]
},
{
"name": "flops_sp",
"unit": {
"base": "FLOPS",
"prefix": "G"
},
"scope": "hwthread",
"aggregation": "sum",
"footprint": "avg",
"timestep": 60,
"peak": 3000,
"normal": 3000,
"caution": 100,
"alert": 50,
"subClusters": [
{"name": "cpu", "remove": true },
{ "name": "cpu_abs2", "remove": true },
{ "name": "fatcpu", "remove": true },
{ "name": "gpu", "remove": true },
{ "name": "fatgpu", "remove": true },
{ "name": "vis", "remove": true },
{ "name": "h200gpu", "remove": true },
{ "name": "tcpu", "peak": 26900, "normal": 22393, "caution": 747, "alert": 374 },
{ "name": "tl40sgpu", "peak": 9500, "normal": 7860, "caution": 262, "alert": 131 },
{ "name": "th200gpu", "peak": 10800, "normal": 8919, "caution": 298, "alert": 149 },
{ "name": "tvis", "peak": 19000, "normal": 15758, "caution": 526, "alert": 263 }
]
},
{
"name": "flops_dp",
"unit": {
"base": "FLOPS",
"prefix": "G"
},
"scope": "hwthread",
"aggregation": "sum",
"footprint": "avg",
"timestep": 60,
"peak": 3000,
"normal": 3000,
"caution": 100,
"alert": 50,
"subClusters": [
{"name": "cpu", "remove": true },
{ "name": "cpu_abs2", "remove": true },
{ "name": "fatcpu", "remove": true },
{ "name": "gpu", "remove": true },
{ "name": "fatgpu", "remove": true },
{ "name": "vis", "remove": true },
{ "name": "h200gpu", "remove": true },
{ "name": "tcpu", "peak": 13500, "normal": 11197, "caution": 373, "alert": 187 },
{ "name": "tl40sgpu", "peak": 4800, "normal": 3930, "caution": 131, "alert": 66 },
{ "name": "th200gpu", "peak": 5400, "normal": 4460, "caution": 149, "alert": 74 },
{ "name": "tvis", "peak": 9500, "normal": 7879, "caution": 263, "alert": 131 }
]
},
{
"name": "node_total_power",
"unit": {
"base": "W"
},
"scope": "node",
"aggregation": "sum",
"timestep": 60,
"peak": 11400,
"normal": 9500,
"caution": 9500,
"alert": 9500,
"subClusters": [
{ "name": "cpu", "peak": 900, "normal": 750, "caution": 750, "alert": 750 },
{ "name": "cpu_abs2", "peak": 900, "normal": 750, "caution": 750, "alert": 750 },
{ "name": "gpu", "peak": 3600, "normal": 3000, "caution": 3000, "alert": 3000 },
{ "name": "fatcpu", "peak": 2000, "normal": 1600, "caution": 1600, "alert": 1600 },
{ "name": "fatgpu", "peak": 11400, "normal": 9500, "caution": 9500, "alert": 9500 },
{ "name": "vis", "peak": 3300, "normal": 2700, "caution": 2700, "alert": 2700 },
{ "name": "h200gpu", "peak": 11400, "normal": 9500, "caution": 9500, "alert": 9500 },
{ "name": "tcpu", "peak": 2400, "normal": 2000, "caution": 2000, "alert": 2000 },
{ "name": "tl40sgpu", "peak": 4200, "normal": 3500, "caution": 3500, "alert": 3500 },
{ "name": "th200gpu", "peak": 11400, "normal": 9500, "caution": 9500, "alert": 9500 },
{ "name": "tvis", "peak": 3600, "normal": 3000, "caution": 3000, "alert": 3000 }
]
},
{
"name": "job_mem_used",
"unit": { "base": "B", "prefix": "G" },
"scope": "hwthread",
"aggregation": "sum",
"timestep": 60,
"peak": 400,
"normal": 384,
"caution": 384,
"alert": 384,
"subClusters": [
{ "name": "fatcpu", "peak": 2600, "normal": 2304, "caution": 2304, "alert": 2304 },
{ "name": "cpu_abs2", "peak": 900, "normal": 768, "caution": 768, "alert": 768 },
{ "name": "fatgpu", "peak": 1500, "normal": 1152, "caution": 1152, "alert": 1152 },
{ "name": "vis", "peak": 1500, "normal": 1152, "caution": 1152, "alert": 1152 },
{ "name": "h200gpu", "peak": 2700, "normal": 2304, "caution": 2304, "alert": 2304 },
{ "name": "tcpu", "peak": 1800, "normal": 1536, "caution": 1536, "alert": 1536 },
{ "name": "tl40sgpu", "peak": 900, "normal": 768, "caution": 768, "alert": 768 },
{ "name": "th200gpu", "peak": 1800, "normal": 1536, "caution": 1536, "alert": 1536 },
{ "name": "tvis", "peak": 900, "normal": 768, "caution": 768, "alert": 768 }
]
}
],
"subClusters": [
{
"name": "cpu",
"processorType": "AMD EPYC 9254 24-Core Processor",
"socketsPerNode": 2,
"coresPerSocket": 24,
"threadsPerCore": 1,
"flopRateScalar": {
"unit": {
"base": "F/s",
"prefix": "G"
},
"value": 517
},
"flopRateSimd": {
"unit": {
"base": "F/s",
"prefix": "G"
},
"value": 3175
},
"memoryBandwidth": {
"unit": {
"base": "B/s",
"prefix": "G"
},
"value": 591
},
"nodes": "cpu[001-284]",
"topology": {
"node": [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47],
"socket": [
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23],
[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47]
],
"memoryDomain": [
[0,1,2,3,4,5],
[6,7,8,9,10,11],
[12,13,14,15,16,17],
[18,19,20,21,22,23],
[24,25,26,27,28,29],
[30,31,32,33,34,35],
[36,37,38,39,40,41],
[42,43,44,45,46,47]
],
"core": [
[0],[1],[2],[3],[4],[5],[6],[7],[8],[9],[10],[11],[12],[13],[14],[15],[16],[17],[18],[19],[20],[21],[22],[23],[24],[25],[26],[27],[28],[29],[30],[31],[32],[33],[34],[35],[36],[37],[38],[39],[40],[41],[42],[43],[44],[45],[46],[47]
]
}
},
{
"name": "cpu_abs2",
"processorType": "AMD EPYC 9254 24-Core Processor",
"socketsPerNode": 2,
"coresPerSocket": 24,
"threadsPerCore": 1,
"flopRateScalar": {
"unit": {
"base": "F/s",
"prefix": "G"
},
"value": 517
},
"flopRateSimd": {
"unit": {
"base": "F/s",
"prefix": "G"
},
"value": 3175
},
"memoryBandwidth": {
"unit": {
"base": "B/s",
"prefix": "G"
},
"value": 591
},
"nodes": "cpu[285-336]",
"topology": {
"node": [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47],
"socket": [
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23],
[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47]
],
"memoryDomain": [
[0,1,2,3,4,5],
[6,7,8,9,10,11],
[12,13,14,15,16,17],
[18,19,20,21,22,23],
[24,25,26,27,28,29],
[30,31,32,33,34,35],
[36,37,38,39,40,41],
[42,43,44,45,46,47]
],
"core": [
[0],[1],[2],[3],[4],[5],[6],[7],[8],[9],[10],[11],[12],[13],[14],[15],[16],[17],[18],[19],[20],[21],[22],[23],[24],[25],[26],[27],[28],[29],[30] ,[31],[32],[33],[34],[35],[36],[37],[38],[39],[40],[41],[42],[43],[44],[45],[46],[47]
]
}
},
{
"name": "fatcpu",
"processorType": "AMD EPYC 9454 48-Core Processor",
"socketsPerNode": 2,
"coresPerSocket": 48,
"threadsPerCore": 1,
"flopRateScalar": {
"unit": {
"base": "F/s",
"prefix": "G"
},
"value": 972
},
"flopRateSimd": {
"unit": {
"base": "F/s",
"prefix": "G"
},
"value": 5809
},
"memoryBandwidth": {
"unit": {
"base": "B/s",
"prefix": "G"
},
"value": 778
},
"nodes": "fatcpu[001-013]",
"topology": {
"node": [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95],
"socket": [
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47],
[48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95]
],
"memoryDomain": [
[0,1,2,3,4,5],
[6,7,8,9,10,11],
[12,13,14,15,16,17],
[18,19,20,21,22,23],
[24,25,26,27,28,29],
[30,31,32,33,34,35],
[36,37,38,39,40,41],
[42,43,44,45,46,47],
[48,49,50,51,52,53],
[54,55,56,57,58,59],
[60,61,62,63,64,65],
[66,67,68,69,70,71],
[72,73,74,75,76,77],
[78,79,80,81,82,83],
[84,85,86,87,88,89],
[90,91,92,93,94,95]
],
"core": [
[0],[1],[2],[3],[4],[5],[6],[7],[8],[9],[10],[11],[12],[13],[14],[15],[16],[17],[18],[19],[20],[21],[22],[23],[24],[25],[26],[27],[28],[29],[30],[31],[32],[33],[34],[35],[36],[37],[38],[39],[40],[41],[42],[43],[44],[45],[46],[47],[48],[49],[50],[51],[52],[53],[54],[55],[56],[57],[58],[59],[60],[61],[62],[63],[64],[65],[66],[67],[68],[69],[70],[71],[72],[73],[74],[75],[76],[77],[78],[79],[80],[81],[82],[83],[84],[85],[86],[87],[88],[89],[90],[91],[92],[93],[94],[95]
]
}
},
{
"name": "gpu",
"processorType": "AMD EPYC 9254 24-Core Processor",
"socketsPerNode": 2,
"coresPerSocket": 24,
"threadsPerCore": 1,
"flopRateScalar": {
"unit": {
"base": "F/s",
"prefix": "G"
},
"value": 508
},
"flopRateSimd": {
"unit": {
"base": "F/s",
"prefix": "G"
},
"value": 3163
},
"memoryBandwidth": {
"unit": {
"base": "B/s",
"prefix": "G"
},
"value": 587
},
"nodes": "gpu[001-020]",
"topology": {
"node": [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47],
"socket": [
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23],
[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47]
],
"memoryDomain": [
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23],
[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47]
],
"core": [
[0],[1],[2],[3],[4],[5],[6],[7],[8],[9],[10],[11],[12],[13],[14],[15],[16],[17],[18],[19],[20],[21],[22],[23],[24],[25],[26],[27],[28],[29],[30],[31],[32],[33],[34],[35],[36],[37],[38],[39],[40],[41],[42],[43],[44],[45],[46],[47]
],
"accelerators": [
{
"id": "00000000:21:00.0",
"type": "Nvidia GPU",
"model": "Nvidia A30"
},
{
"id": "00000000:41:00.0",
"type": "Nvidia GPU",
"model": "Nvidia A30"
},
{
"id": "00000000:A1:00.0",
"type": "Nvidia GPU",
"model": "Nvidia A30"
}
]
}
},
{
"name": "fatgpu",
"processorType": "AMD EPYC 9454 48-Core Processor",
"socketsPerNode": 2,
"coresPerSocket": 48,
"threadsPerCore": 1,
"flopRateScalar": {
"unit": {
"base": "F/s",
"prefix": "G"
},
"value": 962
},
"flopRateSimd": {
"unit": {
"base": "F/s",
"prefix": "G"
},
"value": 5796
},
"memoryBandwidth": {
"unit": {
"base": "B/s",
"prefix": "G"
},
"value": 779
},
"nodes": "fatgpu[001-007]",
"topology": {
"node": [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95],
"socket": [
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47],
[48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95]
],
"memoryDomain": [
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47],
[48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95]
],
"core": [
[0],[1],[2],[3],[4],[5],[6],[7],[8],[9],[10],[11],[12],[13],[14],[15],[16],[17],[18],[19],[20],[21],[22],[23],[24],[25],[26],[27],[28],[29],[30],[31],[32],[33],[34],[35],[36],[37],[38],[39],[40],[41],[42],[43],[44],[45],[46],[47],[48],[49],[50],[51],[52],[53],[54],[55],[56],[57],[58],[59],[60],[61],[62],[63],[64],[65],[66],[67],[68],[69],[70],[71],[72],[73],[74],[75],[76],[77],[78],[79],[80],[81],[82],[83],[84],[85],[86],[87],[88],[89],[90],[91],[92],[93],[94],[95]
],
"accelerators": [
{
"id": "00000000:26:00.0",
"type": "Nvidia GPU",
"model": "Nvidia H100"
},
{
"id": "00000000:2F:00.0",
"type": "Nvidia GPU",
"model": "Nvidia H100"
},
{
"id": "00000000:46:00.0",
"type": "Nvidia GPU",
"model": "Nvidia H100"
},
{
"id": "00000000:54:00.0",
"type": "Nvidia GPU",
"model": "Nvidia H100"
},
{
"id": "00000000:A6:00.0",
"type": "Nvidia GPU",
"model": "Nvidia H100"
},
{
"id": "00000000:AF:00.0",
"type": "Nvidia GPU",
"model": "Nvidia H100"
},
{
"id": "00000000:C6:00.0",
"type": "Nvidia GPU",
"model": "Nvidia H100"
},
{
"id": "00000000:CF:00.0",
"type": "Nvidia GPU",
"model": "Nvidia H100"
}
]
}
},
{
"name": "vis",
"processorType": "AMD EPYC 9254 24-Core Processor",
"socketsPerNode": 2,
"coresPerSocket": 24,
"threadsPerCore": 1,
"flopRateScalar": {
"unit": {
"base": "F/s",
"prefix": "G"
},
"value": 517
},
"flopRateSimd": {
"unit": {
"base": "F/s",
"prefix": "G"
},
"value": 3175
},
"memoryBandwidth": {
"unit": {
"base": "B/s",
"prefix": "G"
},
"value": 588
},
"nodes": "vis[001-003]",
"topology": {
"node": [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47],
"socket": [
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23],
[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47]
],
"memoryDomain": [
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23],
[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47]
],
"core": [
[0],[1],[2],[3],[4],[5],[6],[7],[8],[9],[10],[11],[12],[13],[14],[15],[16],[17],[18],[19],[20],[21],[22],[23],[24],[25],[26],[27],[28],[29],[30],[31],[32],[33],[34],[35],[36],[37],[38],[39],[40],[41],[42],[43],[44],[45],[46],[47]
],
"accelerators": [
{
"id": "00000000:81:00.0",
"type": "Nvidia GPU",
"model": "Nvidia A40"
}
]
}
},
{
"name": "h200gpu",
"processorType": "AMD EPYC 9554 64-Core Processor",
"socketsPerNode": 2,
"coresPerSocket": 64,
"threadsPerCore": 1,
"flopRateScalar": {
"unit": {
"base": "F/s",
"prefix": "G"
},
"value": 1230
},
"flopRateSimd": {
"unit": {
"base": "F/s",
"prefix": "G"
},
"value": 7607
},
"memoryBandwidth": {
"unit": {
"base": "B/s",
"prefix": "G"
},
"value": 790
},
"nodes": "h200gpu001",
"topology": {
"node": [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127],
"socket": [
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63],
[64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127]
],
"memoryDomain": [
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63],
[64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127]
],
"core": [
[0],[1],[2],[3],[4],[5],[6],[7],[8],[9],[10],[11],[12],[13],[14],[15],[16],[17],[18],[19],[20],[21],[22],[23],[24],[25],[26],[27],[28],[29],[30],[31],[32],[33],[34],[35],[36],[37],[38],[39],[40],[41],[42],[43],[44],[45],[46],[47],[48],[49],[50],[51],[52],[53],[54],[55],[56],[57],[58],[59],[60],[61],[62],[63],[64],[65],[66],[67],[68],[69],[70],[71],[72],[73],[74],[75],[76],[77],[78],[79],[80],[81],[82],[83],[84],[85],[86],[87],[88],[89],[90],[91],[92],[93],[94],[95],[96],[97],[98],[99],[100],[101],[102],[103],[104],[105],[106],[107],[108],[109],[110],[111],[112],[113],[114],[115],[116],[117],[118],[119],[120],[121],[122],[123],[124],[125],[126],[127]
],
"accelerators": [
{
"id": "00000000:04:00.0",
"type": "Nvidia GPU",
"model": "Nvidia H200"
},
{
"id": "00000000:24:00.0",
"type": "Nvidia GPU",
"model": "Nvidia H200"
},
{
"id": "00000000:44:00.0",
"type": "Nvidia GPU",
"model": "Nvidia H200"
},
{
"id": "00000000:64:00.0",
"type": "Nvidia GPU",
"model": "Nvidia H200"
},
{
"id": "00000000:84:00.0",
"type": "Nvidia GPU",
"model": "Nvidia H200"
},
{
"id": "00000000:A4:00.0",
"type": "Nvidia GPU",
"model": "Nvidia H200"
},
{
"id": "00000000:C4:00.0",
"type": "Nvidia GPU",
"model": "Nvidia H200"
},
{
"id": "00000000:E4:00.0",
"type": "Nvidia GPU",
"model": "Nvidia H200"
}
]
}
},
{
"name": "tcpu",
"processorType": "AMD EPYC 9645 96-Core Processor",
"socketsPerNode": 2,
"coresPerSocket": 96,
"threadsPerCore": 1,
"flopRateScalar": {
"unit": {
"base": "F/s",
"prefix": "G"
},
"value": 1912
},
"flopRateSimd": {
"unit": {
"base": "F/s",
"prefix": "G"
},
"value": 22393
},
"memoryBandwidth": {
"unit": {
"base": "B/s",
"prefix": "G"
},
"value": 892
},
"nodes": "tcpu[001-047]",
"topology": {
"node": [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191],
"socket": [
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95],
[96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191]
],
"memoryDomain": [
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23],
[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47],
[48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71],
[72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95],
[96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119],
[120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143],
[144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167],
[168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191]
],
"core": [
[0],[1],[2],[3],[4],[5],[6],[7],[8],[9],[10],[11],[12],[13],[14],[15],[16],[17],[18],[19],[20],[21],[22],[23],[24],[25],[26],[27],[28],[29],[30],[31],[32],[33],[34],[35],[36],[37],[38],[39],[40],[41],[42],[43],[44],[45],[46],[47],[48],[49],[50],[51],[52],[53],[54],[55],[56],[57],[58],[59],[60],[61],[62],[63],[64],[65],[66],[67],[68],[69],[70],[71],[72],[73],[74],[75],[76],[77],[78],[79],[80],[81],[82],[83],[84],[85],[86],[87],[88],[89],[90],[91],[92],[93],[94],[95],[96],[97],[98],[99],[100],[101],[102],[103],[104],[105],[106],[107],[108],[109],[110],[111],[112],[113],[114],[115],[116],[117],[118],[119],[120],[121],[122],[123],[124],[125],[126],[127],[128],[129],[130],[131],[132],[133],[134],[135],[136],[137],[138],[139],[140],[141],[142],[143],[144],[145],[146],[147],[148],[149],[150],[151],[152],[153],[154],[155],[156],[157],[158],[159],[160],[161],[162],[163],[164],[165],[166],[167],[168],[169],[170],[171],[172],[173],[174],[175],[176],[177],[178],[179],[180],[181],[182],[183],[184],[185],[186],[187],[188],[189],[190],[191]
]
}
},
{
"name": "th200gpu",
"processorType": "AMD EPYC 9335 32-Core Processor",
"socketsPerNode": 2,
"coresPerSocket": 32,
"threadsPerCore": 1,
"flopRateScalar": {
"unit": {
"base": "F/s",
"prefix": "G"
},
"value": 492
},
"flopRateSimd": {
"unit": {
"base": "F/s",
"prefix": "G"
},
"value": 8919
},
"memoryBandwidth": {
"unit": {
"base": "B/s",
"prefix": "G"
},
"value": 864
},
"nodes": "th200gpu001",
"topology": {
"node": [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63],
"socket": [
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31],
[32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63]
],
"memoryDomain": [
[0,1,2,3,4,5,6,7],
[8,9,10,11,12,13,14,15],
[16,17,18,19,20,21,22,23],
[24,25,26,27,28,29,30,31],
[32,33,34,35,36,37,38,39],
[40,41,42,43,44,45,46,47],
[48,49,50,51,52,53,54,55],
[56,57,58,59,60,61,62,63]
],
"core": [
[0],[1],[2],[3],[4],[5],[6],[7],[8],[9],[10],[11],[12],[13],[14],[15],[16],[17],[18],[19],[20],[21],[22],[23],[24],[25],[26],[27],[28],[29],[30],[31],[32],[33],[34],[35],[36],[37],[38],[39],[40],[41],[42],[43],[44],[45],[46],[47],[48],[49],[50],[51],[52],[53],[54],[55],[56],[57],[58],[59],[60],[61],[62],[63]
],
"accelerators": [
{
"id": "00000000:16:00.0",
"type": "Nvidia GPU",
"model": "Nvidia H200"
},
{
"id": "00000000:66:00.0",
"type": "Nvidia GPU",
"model": "Nvidia H200"
},
{
"id": "00000000:96:00.0",
"type": "Nvidia GPU",
"model": "Nvidia H200"
},
{
"id": "00000000:E6:00.0",
"type": "Nvidia GPU",
"model": "Nvidia H200"
}
]
}
},
{
"name": "tl40sgpu",
"processorType": "AMD EPYC 9535 64-Core Processor",
"socketsPerNode": 1,
"coresPerSocket": 64,
"threadsPerCore": 1,
"flopRateScalar": {
"unit": {
"base": "F/s",
"prefix": "G"
},
"value": 590
},
"flopRateSimd": {
"unit": {
"base": "F/s",
"prefix": "G"
},
"value": 7860
},
"memoryBandwidth": {
"unit": {
"base": "B/s",
"prefix": "G"
},
"value": 576
},
"nodes": "tl40sgpu[001-010]",
"topology": {
"node": [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63],
"socket": [
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63]
],
"memoryDomain": [
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15],
[16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31],
[32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47],
[48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63]
],
"core": [
[0],[1],[2],[3],[4],[5],[6],[7],[8],[9],[10],[11],[12],[13],[14],[15],[16],[17],[18],[19],[20],[21],[22],[23],[24],[25],[26],[27],[28],[29],[30],[31],[32],[33],[34],[35],[36],[37],[38],[39],[40],[41],[42],[43],[44],[45],[46],[47],[48],[49],[50],[51],[52],[53],[54],[55],[56],[57],[58],[59],[60],[61],[62],[63]
],
"accelerators": [
{
"id": "00000000:01:00.0",
"type": "Nvidia GPU",
"model": "Nvidia L40S"
},
{
"id": "00000000:21:00.0",
"type": "Nvidia GPU",
"model": "Nvidia L40S"
},
{
"id": "00000000:C1:00.0",
"type": "Nvidia GPU",
"model": "Nvidia L40S"
},
{
"id": "00000000:E1:00.0",
"type": "Nvidia GPU",
"model": "Nvidia L40S"
}
]
}
},
{
"name": "tvis",
"processorType": "AMD EPYC 9535 64-Core Processor",
"socketsPerNode": 2,
"coresPerSocket": 64,
"threadsPerCore": 1,
"flopRateScalar": {
"unit": {
"base": "F/s",
"prefix": "G"
},
"value": 1296
},
"flopRateSimd": {
"unit": {
"base": "F/s",
"prefix": "G"
},
"value": 15758
},
"memoryBandwidth": {
"unit": {
"base": "B/s",
"prefix": "G"
},
"value": 1148
},
"nodes": "tvis[001-002]",
"topology": {
"node": [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127],
"socket": [
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63],
[64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127]
],
"memoryDomain": [
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15],
[16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31],
[32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47],
[48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63],
[64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79],
[80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95],
[96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111],
[112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127]
],
"core": [
[0],[1],[2],[3],[4],[5],[6],[7],[8],[9],[10],[11],[12],[13],[14],[15],[16],[17],[18],[19],[20],[21],[22],[23],[24],[25],[26],[27],[28],[29],[30],[31],[32],[33],[34],[35],[36],[37],[38],[39],[40],[41],[42],[43],[44],[45],[46],[47],[48],[49],[50],[51],[52],[53],[54],[55],[56],[57],[58],[59],[60],[61],[62],[63],[64],[65],[66],[67],[68],[69],[70],[71],[72],[73],[74],[75],[76],[77],[78],[79],[80],[81],[82],[83],[84],[85],[86],[87],[88],[89],[90],[91],[92],[93],[94],[95],[96],[97],[98],[99],[100],[101],[102],[103],[104],[105],[106],[107],[108],[109],[110],[111],[112],[113],[114],[115],[116],[117],[118],[119],[120],[121],[122],[123],[124],[125],[126],[127]
],
"accelerators": [
{
"id": "00000000:11:00.0",
"type": "Nvidia GPU",
"model": "Nvidia RTX 4500 Ada"
}
]
}
}
]
}
cc-metric-collector/config.json
Inhalt anzeigen
{
"sinks": "./sinks.json",
"collectors" : "./collectors.json",
"receivers" : "./receivers.json",
"router" : "./router.json",
"interval": "60s",
"duration": "10s"
}
cc-metric-collector/router.json
Inhalt anzeigen
{
"add_tags": [
{ "key": "cluster", "value": "elysium", "if": "*" }
],
"interval_timestamp": false,
"num_cache_intervals": 0,
"process_messages": {
"stage_order": ["add_tag","rename","rename_if","drop_by_type","drop_by_name","drop_if","change_unit_prefix","normalize_unit"],
"hostname_tag": "hostname",
"rename_messages": {
"load_one": "cpu_load",
"net_bytes_in_bw": "net_bytes_in",
"net_bytes_out_bw": "net_bytes_out",
"net_pkts_in_bw": "net_pkts_in",
"net_pkts_out_bw": "net_pkts_out",
"ib_recv_bw": "ib_recv",
"ib_xmit_bw": "ib_xmit",
"ib_recv_pkts_bw": "ib_recv_pkts",
"ib_xmit_pkts_bw": "ib_xmit_pkts",
"lustre_open_diff": "lustre_open",
"lustre_close_diff": "lustre_close",
"lustre_statfs_diff": "lustre_statfs",
"nv_util": "acc_utilization",
"nv_fb_mem_used": "acc_mem_used",
"nv_power_usage": "acc_power",
"nv_mem_util": "acc_mem_util",
"nfsio_nread_bw": "nfsio_nread",
"nfsio_nwrite_bw": "nfsio_nwrite",
"numastats_interleave_hit_rate": "numastats_interleave_hit",
"numastats_local_node_rate": "numastats_local_node",
"numastats_numa_foreign_rate": "numastats_numa_foreign",
"numastats_numa_hit_rate": "numastats_numa_hit",
"numastats_numa_miss_rate": "numastats_numa_miss",
"numastats_other_node_rate": "numastats_other_node"
},
"rename_messages_if": {},
"drop_messages_if": [
"(tag.hostname startsWith 'cpu' || tag.hostname startsWith 'fatcpu' || tag.hostname startsWith 'gpu' || tag.hostname startsWith 'vis') && (name == 'disk_free' || name == 'io_reads' || name == 'io_writes') && !(tag.device == '/dev/nvme0n1p4' || tag.device == 'nvme0n1p4')",
"tag.hostname startsWith 'fatgpu' && (name == 'disk_free' || name == 'io_reads' || name == 'io_writes') && !(tag.device == '/dev/md127' || tag.device == 'md127')",
"tag.hostname startsWith 'tl40sgpu' && (name == 'disk_free' || name == 'io_reads' || name == 'io_writes') && !(tag.device == '/dev/nvme0n1p1' || tag.device == 'nvme0n1p1')",
"tag.hostname startsWith 'tvis' && (name == 'disk_free' || name == 'io_reads' || name == 'io_writes') && !(tag.device == '/dev/md125' || tag.device == 'md125')",
"(tag.hostname startsWith 'tcpu' || tag.hostname == 'th200gpu001') && (name == 'disk_free' || name == 'io_reads' || name == 'io_writes') && !(tag.device == '/dev/sda4' || tag.device == 'sda4')",
"tag.hostname == 'h200gpu001' && (name == 'disk_free' || name == 'io_reads' || name == 'io_writes') && !(tag.device == '/dev/nvme0n1p1' || tag.device == 'nvme0n1p1')",
"name == 'nfsio_nread' && !(tag.stypeid matches 'home:/home')",
"name == 'nfsio_nwrite' && !(tag.stypeid matches 'home:/home')",
"messagetype == 'metric' && !(name in ['cpu_load','cpu_load_core','cpu_user','mem_used','numastats_interleave_hit','numastats_local_node','numastats_numa_foreign','numastats_numa_hit','numastats_numa_miss','numastats_other_node','disk_free','io_reads','io_writes','lustre_read_bw','lustre_write_bw','lustre_open','lustre_close','lustre_statfs','net_bytes_in','net_bytes_out','net_pkts_in','net_pkts_out','ib_recv','ib_xmit','ib_recv_pkts','ib_xmit_pkts','nfs4_open','nfs4_close','nfsio_nread','nfsio_nwrite','nread','nwrite','acc_utilization','acc_mem_used','acc_power','acc_mem_util','nv_compute_processes','mem_bw','flops_any','clock','ipc','core_power','node_total_power','job_mem_used','flops_bf16','flops_sp','flops_dp'])"
],
"change_unit_prefix": {
"name == 'mem_used'": "G",
"name == 'acc_mem_used'": "G",
"name == 'lustre_read_bw'": "M",
"name == 'lustre_write_bw'": "M",
"name == 'ib_recv'": "M",
"name == 'ib_xmit'": "M",
"name == 'disk_free'": "G",
"name == 'net_bytes_in'": "M",
"name == 'net_bytes_out'": "M",
"name == 'nfsio_nread'": "M",
"name == 'nfsio_nwrite'": "M",
"name == 'job_mem_used'": "G"
}
},
"normalize_units": true
}
cc-metric-collector/sinks.json
Inhalt anzeigen
{
"elysiumstore": {
"type": "http",
"url": "https://jobmon.hpc.rub.de/api/write/?cluster=elysium",
"jwt": "",
"precision": "s",
"meta_as_tags": [
"unit"
],
"idle_connection_timeout": "60s",
"max_retries": 10,
"timeout": "10s"
}
}