kubelet 启动失败
# 问题分析
systemctl status kubelet 发现启动失败 如下
× kubelet.service - Kubernetes Kubelet
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; preset: disabled)
Active: failed (Result: exit-code) since Sun 2024-12-01 19:35:01 CST; 8min ago
Duration: 61ms
Docs: https://github.com/kubernetes/kubernetes
Process: 857774 ExecStart=/usr/local/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --config=/etc/kubernetes/kubelet-conf.yml --container-runtime-endpoint=unix:///run/containerd/containerd.sock --node-labels=node.kubernetes.io/node= (code=exited, status=2)
Main PID: 857774 (code=exited, status=2)
CPU: 61ms
12月 01 19:35:01 k8s-master-01 kubelet[857774]: google.golang.org/grpc@v1.56.3/internal/transport/http2_client.go:449 +0x22fb
12月 01 19:35:01 k8s-master-01 kubelet[857774]: runtime stack:
12月 01 19:35:01 k8s-master-01 kubelet[857774]: runtime.throw({0x4550fdf?, 0x0?})
12月 01 19:35:01 k8s-master-01 kubelet[857774]: runtime/panic.go:1023 +0x5c fp=0x7f9825ffac20 sp=0x7f9825ffabf0 pc=0x43fe9c
12月 01 19:35:01 k8s-master-01 kubelet[857774]: runtime.newstack()
12月 01 19:35:01 k8s-master-01 kubelet[857774]: runtime/stack.go:1047 +0x74d fp=0x7f9825ffadd0 sp=0x7f9825ffac20 pc=0x45c3cd
12月 01 19:35:01 k8s-master-01 kubelet[857774]: runtime.morestack()
12月 01 19:35:01 k8s-master-01 kubelet[857774]: runtime/asm_amd64.s:616 +0x7a fp=0x7f9825ffadd8 sp=0x7f9825ffadd0 pc=0x478f5a
12月 01 19:35:01 k8s-master-01 systemd[1]: kubelet.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
12月 01 19:35:01 k8s-master-01 systemd[1]: kubelet.service: Failed with result 'exit-code'.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
通过 journalctl 查看 kubelet 的日志，获取更多信息：
journalctl -u kubelet -b
1
2月 01 19:34:05 k8s-master-01 systemd[1]: Started Kubernetes Kubelet.
12月 01 19:34:05 k8s-master-01 kubelet[857429]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
12月 01 19:34:05 k8s-master-01 kubelet[857429]: I1201 19:34:05.320277 857429 server.go:467] "Kubelet version" kubeletVersion="v1.28.12"
12月 01 19:34:05 k8s-master-01 kubelet[857429]: I1201 19:34:05.320322 857429 server.go:469] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
12月 01 19:34:05 k8s-master-01 kubelet[857429]: I1201 19:34:05.320501 857429 server.go:895] "Client rotation is on, will bootstrap in background"
12月 01 19:34:05 k8s-master-01 kubelet[857429]: I1201 19:34:05.321619 857429 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
12月 01 19:34:05 k8s-master-01 kubelet[857429]: I1201 19:34:05.322215 857429 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.pem"
12月 01 19:34:05 k8s-master-01 kubelet[857429]: fatal error: sync: unlock of unlocked mutex
12月 01 19:34:05 k8s-master-01 kubelet[857429]: goroutine 158 [running]:
12月 01 19:34:05 k8s-master-01 kubelet[857429]: sync.fatal({0x4556c6f?, 0xc?})
12月 01 19:34:05 k8s-master-01 kubelet[857429]: runtime/panic.go:1007 +0x18
12月 01 19:34:05 k8s-master-01 kubelet[857429]: sync.(*Mutex).unlockSlow(0xc000bd4c48, 0x4b3915f)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: sync/mutex.go:229 +0x35
12月 01 19:34:05 k8s-master-01 kubelet[857429]: sync.(*Mutex).Unlock(...)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: sync/mutex.go:223
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc/internal/transport.(*recvBuffer).put(0xc000bd4c40, {0x0?, {0x4b39160?, 0x6d251b0?}})
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc@v1.56.3/internal/transport/transport.go:122 +0x207
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc/internal/transport.(*Stream).write(...)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc@v1.56.3/internal/transport/transport.go:488
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc/internal/transport.(*http2Client).closeStream(0xc0007beb48, 0xc00084d560, {0x4b39160?, 0x6d251b0?}, 0x0, 0x0, 0x0?, 0x0?, 0x1)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc@v1.56.3/internal/transport/http2_client.go:904 +0xcd
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc/internal/transport.(*http2Client).operateHeaders(0xc0007beb48, 0xc000913e00)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc@v1.56.3/internal/transport/http2_client.go:1559 +0x1f65
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0007beb48, 0xc00079f920)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc@v1.56.3/internal/transport/http2_client.go:1628 +0x43b
12月 01 19:34:05 k8s-master-01 kubelet[857429]: created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 170
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc@v1.56.3/internal/transport/http2_client.go:397 +0x1d53
12月 01 19:34:05 k8s-master-01 kubelet[857429]: goroutine 1 [runnable]:
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc00084d560)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc@v1.56.3/internal/transport/transport.go:331 +0x7c
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc@v1.56.3/internal/transport/transport.go:346
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc.(*csAttempt).recvMsg(0xc000813ad0, {0x4387a00, 0xc0004b6320}, 0xc000767dc8?)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc@v1.56.3/stream.go:1066 +0xc9
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x6dfff00?)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc@v1.56.3/stream.go:917 +0x1f
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc.(*clientStream).withRetry(0xc00084d320, 0xc000767e18, 0xc000767e08)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc@v1.56.3/stream.go:768 +0x13a
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc.(*clientStream).RecvMsg(0xc00084d320, {0x4387a00?, 0xc0004b6320?})
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc@v1.56.3/stream.go:916 +0x11c
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc.invoke({0x4b6acb0?, 0xc0009121e0?}, {0x456f45e?, 0x4?}, {0x42ccc80, 0xc000012138}, {0x4387a00, 0xc0004b6320}, 0x0?, {0xc0008e2ad0, ...})
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc@v1.56.3/call.go:78 +0xcb
12月 01 19:34:05 k8s-master-01 kubelet[857429]: go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryClientInterceptor.func1({0x4b6ad58, 0xc000215d50}, {0x456f45e, 0x22}, {0x42ccc80, 0xc000012138}, {0x4387a00, 0xc0004b6320}, 0xc000a22408, 0x4703d28, ...)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.35.0/interceptor.go:108 +0x828
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc.(*ClientConn).Invoke(0xc000a22408, {0x4b6ad58, 0xc000215d50}, {0x456f45e, 0x22}, {0x42ccc80, 0xc000012138}, {0x4387a00, 0xc0004b6320}, {0x0, ...})
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc@v1.56.3/call.go:40 +0x23c
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/cri-api/pkg/apis/runtime/v1.(*runtimeServiceClient).Version(0xc0006ef190, {0x4b6ad58, 0xc000215d50}, 0xc000012138, {0x0, 0x0, 0x0})
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/cri-api@v0.0.0/pkg/apis/runtime/v1/api.pb.go:10545 +0xc5
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/kubernetes/pkg/kubelet/cri/remote.(*remoteRuntimeService).validateServiceConnection(0xc0008f0be0, {0x4b6ad58, 0xc000215d50}, 0xc000a22408, {0x7ffc9c82ceeb, 0x26})
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/kubernetes/pkg/kubelet/cri/remote/remote_runtime.go:139 +0x143
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/kubernetes/pkg/kubelet/cri/remote.NewRemoteRuntimeService({0x7ffc9c82ceeb, 0x26}, 0x1bf08eb000, {0x4b392a0, 0x6dfff00})
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/kubernetes/pkg/kubelet/cri/remote/remote_runtime.go:126 +0xba5
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/kubernetes/pkg/kubelet.PreInitRuntimeService(0xc000948b50, 0xc00047e300)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/kubernetes/pkg/kubelet/kubelet.go:322 +0x85
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/kubernetes/cmd/kubelet/app.run({0x4b6ace8, 0xc0004381e0}, 0xc000948a08, 0xc00047e300, {0x7f38546010d8, 0xc000225700})
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/kubernetes/cmd/kubelet/app/server.go:681 +0xbe7
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/kubernetes/cmd/kubelet/app.Run({0x4b6ace8, 0xc0004381e0}, 0xc000948a08, 0xc00047e300, {0x7f38546010d8, 0xc000225700})
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/kubernetes/cmd/kubelet/app/server.go:474 +0x333
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/kubernetes/cmd/kubelet/app.NewKubeletCommand.func1(0xc00037c908, {0xc0000741f0, 0x5, 0x5})
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/kubernetes/cmd/kubelet/app/server.go:283 +0xc29
12月 01 19:34:05 k8s-master-01 kubelet[857429]: github.com/spf13/cobra.(*Command).execute(0xc00037c908, {0xc0000741f0, 0x5, 0x5})
12月 01 19:34:05 k8s-master-01 kubelet[857429]: github.com/spf13/cobra@v1.7.0/command.go:940 +0x882
12月 01 19:34:05 k8s-master-01 kubelet[857429]: github.com/spf13/cobra.(*Command).ExecuteC(0xc00037c908)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5
12月 01 19:34:05 k8s-master-01 kubelet[857429]: github.com/spf13/cobra.(*Command).Execute(...)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: github.com/spf13/cobra@v1.7.0/command.go:992
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/component-base/cli.run(0xc00037c908)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/component-base@v0.0.0/cli/run.go:146 +0x290
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/component-base/cli.Run(0xc0000061c0?)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/component-base@v0.0.0/cli/run.go:46 +0x17
12月 01 19:34:05 k8s-master-01 kubelet[857429]: main.main()
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/kubernetes/cmd/kubelet/kubelet.go:36 +0x18
12月 01 19:34:05 k8s-master-01 kubelet[857429]: goroutine 9 [select]:
12月 01 19:34:05 k8s-master-01 kubelet[857429]: go.opencensus.io/stats/view.(*worker).start(0xc000167980)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f
12月 01 19:34:05 k8s-master-01 kubelet[857429]: created by go.opencensus.io/stats/view.init.0 in goroutine 1
12月 01 19:34:05 k8s-master-01 kubelet[857429]: go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d
12月 01 19:34:05 k8s-master-01 kubelet[857429]: goroutine 151 [select]:
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/klog/v2.(*flushDaemon).run.func1()
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/klog/v2@v2.100.1/klog.go:1141 +0x117
12月 01 19:34:05 k8s-master-01 kubelet[857429]: created by k8s.io/klog/v2.(*flushDaemon).run in goroutine 1
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/klog/v2@v2.100.1/klog.go:1137 +0x171
12月 01 19:34:05 k8s-master-01 kubelet[857429]: goroutine 30 [syscall]:
12月 01 19:34:05 k8s-master-01 kubelet[857429]: os/signal.signal_recv()
12月 01 19:34:05 k8s-master-01 kubelet[857429]: runtime/sigqueue.go:152 +0x29
12月 01 19:34:05 k8s-master-01 kubelet[857429]: os/signal.loop()
12月 01 19:34:05 k8s-master-01 kubelet[857429]: os/signal/signal_unix.go:23 +0x13
12月 01 19:34:05 k8s-master-01 kubelet[857429]: created by os/signal.Notify.func1.1 in goroutine 1
12月 01 19:34:05 k8s-master-01 kubelet[857429]: os/signal/signal.go:151 +0x1f
12月 01 19:34:05 k8s-master-01 kubelet[857429]: goroutine 31 [chan receive]:
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apiserver/pkg/server.SetupSignalContext.func1()
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apiserver@v0.0.0/pkg/server/signal.go:48 +0x25
12月 01 19:34:05 k8s-master-01 kubelet[857429]: created by k8s.io/apiserver/pkg/server.SetupSignalContext in goroutine 1
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apiserver@v0.0.0/pkg/server/signal.go:47 +0xda
12月 01 19:34:05 k8s-master-01 kubelet[857429]: goroutine 162 [select]:
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0006fbb80, {0x4b3ae00, 0xc000a04300}, 0x1, 0xc0000d2300)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:238 +0x12c
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0006fbb80, 0x2540be400, 0x0, 0x1, 0xc0000d2300)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:204 +0x7f
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery/pkg/util/wait.Until(...)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:161
12月 01 19:34:05 k8s-master-01 kubelet[857429]: created by k8s.io/kubernetes/pkg/kubelet/certificate.addCertRotation in goroutine 1
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/kubernetes/pkg/kubelet/certificate/transport.go:156 +0x309
12月 01 19:34:05 k8s-master-01 kubelet[857429]: goroutine 163 [select]:
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/client-go/util/certificate.(*manager).Start.func1()
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/client-go@v0.0.0/util/certificate/certificate_manager.go:415 +0x212
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:226 +0x33
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000881c08, {0x4b3ae00, 0xc000a04630}, 0x1, 0xc0008a0ae0)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:227 +0xaf
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000881c08, 0x3b9aca00, 0x0, 0x1, 0xc0008a0ae0)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:204 +0x7f
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery/pkg/util/wait.Until(...)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:161
12月 01 19:34:05 k8s-master-01 kubelet[857429]: created by k8s.io/client-go/util/certificate.(*manager).Start in goroutine 1
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/client-go@v0.0.0/util/certificate/certificate_manager.go:407 +0x167
12月 01 19:34:05 k8s-master-01 kubelet[857429]: goroutine 164 [chan receive]:
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc0008962a0)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/client-go@v0.0.0/util/workqueue/queue.go:314 +0x9c
12月 01 19:34:05 k8s-master-01 kubelet[857429]: created by k8s.io/client-go/util/workqueue.newQueue in goroutine 1
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/client-go@v0.0.0/util/workqueue/queue.go:106 +0x1a5
12月 01 19:34:05 k8s-master-01 kubelet[857429]: goroutine 165 [select]:
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc0008964e0)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/client-go@v0.0.0/util/workqueue/delaying_queue.go:276 +0x2ff
12月 01 19:34:05 k8s-master-01 kubelet[857429]: created by k8s.io/client-go/util/workqueue.newDelayingQueue in goroutine 1
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/client-go@v0.0.0/util/workqueue/delaying_queue.go:113 +0x205
12月 01 19:34:05 k8s-master-01 kubelet[857429]: goroutine 166 [select]:
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/kubernetes/cmd/kubelet/app.BuildAuthn.func1.1()
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/kubernetes/cmd/kubelet/app/auth.go:103 +0x67
12月 01 19:34:05 k8s-master-01 kubelet[857429]: created by k8s.io/kubernetes/cmd/kubelet/app.BuildAuthn.func1 in goroutine 1
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/kubernetes/cmd/kubelet/app/auth.go:102 +0xac
12月 01 19:34:05 k8s-master-01 kubelet[857429]: goroutine 167 [chan receive]:
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).Run(0xc000896540, {0x4b6ace8, 0xc0006fbe50}, 0x0?)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apiserver@v0.0.0/pkg/server/dynamiccertificates/dynamic_cafile_content.go:170 +0x375
12月 01 19:34:05 k8s-master-01 kubelet[857429]: created by k8s.io/kubernetes/cmd/kubelet/app.BuildAuthn.func1 in goroutine 1
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/kubernetes/cmd/kubelet/app/auth.go:110 +0x110
12月 01 19:34:05 k8s-master-01 kubelet[857429]: goroutine 168 [select]:
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0008f0a40, {0x4b6ace8, 0xc0004b6000})
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc@v1.56.3/internal/grpcsync/callback_serializer.go:83 +0x112
12月 01 19:34:05 k8s-master-01 kubelet[857429]: created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc@v1.56.3/internal/grpcsync/callback_serializer.go:55 +0x11a
12月 01 19:34:05 k8s-master-01 kubelet[857429]: goroutine 169 [select]:
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0008f0aa0, {0x4b6ace8, 0xc0004b60a0})
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc@v1.56.3/internal/grpcsync/callback_serializer.go:83 +0x112
12月 01 19:34:05 k8s-master-01 kubelet[857429]: created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc@v1.56.3/internal/grpcsync/callback_serializer.go:55 +0x11a
12月 01 19:34:05 k8s-master-01 kubelet[857429]: goroutine 154 [sync.Cond.Wait]:
12月 01 19:34:05 k8s-master-01 kubelet[857429]: sync.runtime_notifyListWait(0xc0008a77d0, 0x0)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: runtime/sema.go:569 +0x159
12月 01 19:34:05 k8s-master-01 kubelet[857429]: sync.(*Cond).Wait(0x3882c40?)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: sync/cond.go:70 +0x85
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/client-go/util/workqueue.(*Type).Get(0xc0008962a0)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/client-go@v0.0.0/util/workqueue/queue.go:200 +0x93
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).processNextWorkItem(0xc000896540)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apiserver@v0.0.0/pkg/server/dynamiccertificates/dynamic_cafile_content.go:225 +0x47
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).runWorker(...)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apiserver@v0.0.0/pkg/server/dynamiccertificates/dynamic_cafile_content.go:220
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:226 +0x33
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00041a070, {0x4b3ae00, 0xc000b7e1e0}, 0x1, 0xc0008010e0)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:227 +0xaf
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00041a070, 0x3b9aca00, 0x0, 0x1, 0xc0008010e0)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:204 +0x7f
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery/pkg/util/wait.Until(...)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:161
12月 01 19:34:05 k8s-master-01 kubelet[857429]: created by k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).Run in goroutine 167
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apiserver@v0.0.0/pkg/server/dynamiccertificates/dynamic_cafile_content.go:161 +0x289
12月 01 19:34:05 k8s-master-01 kubelet[857429]: goroutine 155 [select]:
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).watchCAFile(0xc000896540, 0xc0008010e0)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apiserver@v0.0.0/pkg/server/dynamiccertificates/dynamic_cafile_content.go:190 +0x2d5
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).Run.func1()
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apiserver@v0.0.0/pkg/server/dynamiccertificates/dynamic_cafile_content.go:165 +0x32
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:226 +0x33
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000b93360, {0x4b3ae00, 0xc0009edce0}, 0x1, 0xc0008010e0)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:227 +0xaf
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000b93360, 0xdf8475800, 0x0, 0x1, 0xc0008010e0)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:204 +0x7f
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery/pkg/util/wait.Until(...)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:161
12月 01 19:34:05 k8s-master-01 kubelet[857429]: created by k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).Run in goroutine 167
12月 01 19:34:05 k8s-master-01 kubelet[857429]: k8s.io/apiserver@v0.0.0/pkg/server/dynamiccertificates/dynamic_cafile_content.go:164 +0x358
12月 01 19:34:05 k8s-master-01 kubelet[857429]: goroutine 156 [IO wait]:
12月 01 19:34:05 k8s-master-01 kubelet[857429]: internal/poll.runtime_pollWait(0x7f389d4987f0, 0x72)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: runtime/netpoll.go:345 +0x85
12月 01 19:34:05 k8s-master-01 kubelet[857429]: internal/poll.(*pollDesc).wait(0xc00079f680?, 0xc000ba3e6b?, 0x1)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: internal/poll/fd_poll_runtime.go:84 +0x27
12月 01 19:34:05 k8s-master-01 kubelet[857429]: internal/poll.(*pollDesc).waitRead(...)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: internal/poll/fd_poll_runtime.go:89
12月 01 19:34:05 k8s-master-01 kubelet[857429]: internal/poll.(*FD).Read(0xc00079f680, {0xc000ba3e6b, 0x10000, 0x10000})
12月 01 19:34:05 k8s-master-01 kubelet[857429]: internal/poll/fd_unix.go:164 +0x27a
12月 01 19:34:05 k8s-master-01 kubelet[857429]: os.(*File).read(...)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: os/file_posix.go:29
12月 01 19:34:05 k8s-master-01 kubelet[857429]: os.(*File).Read(0xc000770ca0, {0xc000ba3e6b?, 0x0?, 0x0?})
12月 01 19:34:05 k8s-master-01 kubelet[857429]: os/file.go:118 +0x52
12月 01 19:34:05 k8s-master-01 kubelet[857429]: github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc000438280)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: github.com/fsnotify/fsnotify@v1.6.0/backend_inotify.go:356 +0xcf
12月 01 19:34:05 k8s-master-01 kubelet[857429]: created by github.com/fsnotify/fsnotify.NewWatcher in goroutine 155
12月 01 19:34:05 k8s-master-01 kubelet[857429]: github.com/fsnotify/fsnotify@v1.6.0/backend_inotify.go:150 +0x186
12月 01 19:34:05 k8s-master-01 kubelet[857429]: goroutine 159 [runnable]:
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000438460, 0x1)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc@v1.56.3/internal/transport/controlbuf.go:418 +0x113
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000026e00)
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc@v1.56.3/internal/transport/controlbuf.go:552 +0x86
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc@v1.56.3/internal/transport/http2_client.go:451 +0x85
12月 01 19:34:05 k8s-master-01 kubelet[857429]: created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 170
12月 01 19:34:05 k8s-master-01 kubelet[857429]: google.golang.org/grpc@v1.56.3/internal/transport/http2_client.go:449 +0x22fb
12月 01 19:34:05 k8s-master-01 systemd[1]: kubelet.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
12月 01 19:34:05 k8s-master-01 systemd[1]: kubelet.service: Failed with result 'exit-code'.
12月 01 19:35:01 k8s-master-01 systemd[1]: Started Kubernetes Kubelet.
12月 01 19:35:01 k8s-master-01 kubelet[857774]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
12月 01 19:35:01 k8s-master-01 kubelet[857774]: I1201 19:35:01.915646 857774 server.go:467] "Kubelet version" kubeletVersion="v1.28.12"
12月 01 19:35:01 k8s-master-01 kubelet[857774]: I1201 19:35:01.915697 857774 server.go:469] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
12月 01 19:35:01 k8s-master-01 kubelet[857774]: I1201 19:35:01.915829 857774 server.go:895] "Client rotation is on, will bootstrap in background"
12月 01 19:35:01 k8s-master-01 kubelet[857774]: I1201 19:35:01.916834 857774 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
12月 01 19:35:01 k8s-master-01 kubelet[857774]: I1201 19:35:01.917375 857774 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.pem"
12月 01 19:35:01 k8s-master-01 kubelet[857774]: fatal error: sync: unlock of unlocked mutex
12月 01 19:35:01 k8s-master-01 kubelet[857774]: runtime: newstack sp=0xc000bfd860 stack=[0xc000c00000, 0xc000c10000]
12月 01 19:35:01 k8s-master-01 kubelet[857774]: morebuf={pc:0x170eb55 sp:0xc000bfd870 lr:0x0}
12月 01 19:35:01 k8s-master-01 kubelet[857774]: sched={pc:0x4559d6 sp:0xc000bfd868 lr:0x0 ctxt:0x0}
12月 01 19:35:01 k8s-master-01 kubelet[857774]: runtime: gp=0xc0000061c0, goid=1, gp->status=0x2
12月 01 19:35:01 k8s-master-01 kubelet[857774]: runtime: split stack overflow: 0xc000bfd860 < 0xc000c00000
12月 01 19:35:01 k8s-master-01 kubelet[857774]: fatal error: runtime: split stack overflow
12月 01 19:35:01 k8s-master-01 kubelet[857774]: goroutine 152 [running]:
12月 01 19:35:01 k8s-master-01 kubelet[857774]: sync.fatal({0x4556c6f?, 0x41337e?})
12月 01 19:35:01 k8s-master-01 kubelet[857774]: runtime/panic.go:1007 +0x18
12月 01 19:35:01 k8s-master-01 kubelet[857774]: sync.(*Mutex).unlockSlow(0xc00079ec88, 0x4b3915f)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: sync/mutex.go:229 +0x35
12月 01 19:35:01 k8s-master-01 kubelet[857774]: sync.(*Mutex).Unlock(...)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: sync/mutex.go:223
12月 01 19:35:01 k8s-master-01 kubelet[857774]: google.golang.org/grpc/internal/transport.(*recvBuffer).put(0xc00079ec80, {0x0?, {0x4b39160?, 0x6d251b0?}})
12月 01 19:35:01 k8s-master-01 kubelet[857774]: google.golang.org/grpc@v1.56.3/internal/transport/transport.go:122 +0x207
12月 01 19:35:01 k8s-master-01 kubelet[857774]: google.golang.org/grpc/internal/transport.(*Stream).write(...)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: google.golang.org/grpc@v1.56.3/internal/transport/transport.go:488
12月 01 19:35:01 k8s-master-01 kubelet[857774]: google.golang.org/grpc/internal/transport.(*http2Client).closeStream(0xc000962008, 0xc00011a900, {0x4b39160?, 0x6d251b0?}, 0x0, 0x0, 0x0?, 0x0?, 0x1)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: google.golang.org/grpc@v1.56.3/internal/transport/http2_client.go:904 +0xcd
12月 01 19:35:01 k8s-master-01 kubelet[857774]: google.golang.org/grpc/internal/transport.(*http2Client).operateHeaders(0xc000962008, 0xc000bed350)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: google.golang.org/grpc@v1.56.3/internal/transport/http2_client.go:1559 +0x1f65
12月 01 19:35:01 k8s-master-01 kubelet[857774]: google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000962008, 0xc00093e360)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: google.golang.org/grpc@v1.56.3/internal/transport/http2_client.go:1628 +0x43b
12月 01 19:35:01 k8s-master-01 kubelet[857774]: created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 169
12月 01 19:35:01 k8s-master-01 kubelet[857774]: google.golang.org/grpc@v1.56.3/internal/transport/http2_client.go:397 +0x1d53
12月 01 19:35:01 k8s-master-01 kubelet[857774]: goroutine 1 gp=0xc0000061c0 m=10 mp=0xc000546008 [running]:
12月 01 19:35:01 k8s-master-01 kubelet[857774]: goroutine running on other thread; stack unavailable
12月 01 19:35:01 k8s-master-01 kubelet[857774]: goroutine 6 [select]:
12月 01 19:35:01 k8s-master-01 kubelet[857774]: go.opencensus.io/stats/view.(*worker).start(0xc000523380)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f
12月 01 19:35:01 k8s-master-01 kubelet[857774]: created by go.opencensus.io/stats/view.init.0 in goroutine 1
12月 01 19:35:01 k8s-master-01 kubelet[857774]: go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d
12月 01 19:35:01 k8s-master-01 kubelet[857774]: goroutine 139 [select]:
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/klog/v2.(*flushDaemon).run.func1()
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/klog/v2@v2.100.1/klog.go:1141 +0x117
12月 01 19:35:01 k8s-master-01 kubelet[857774]: created by k8s.io/klog/v2.(*flushDaemon).run in goroutine 1
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/klog/v2@v2.100.1/klog.go:1137 +0x171
12月 01 19:35:01 k8s-master-01 kubelet[857774]: goroutine 148 [sync.Cond.Wait]:
12月 01 19:35:01 k8s-master-01 kubelet[857774]: sync.runtime_notifyListWait(0xc000b46890, 0x0)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: runtime/sema.go:569 +0x159
12月 01 19:35:01 k8s-master-01 kubelet[857774]: sync.(*Cond).Wait(0x3882c40?)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: sync/cond.go:70 +0x85
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/client-go/util/workqueue.(*Type).Get(0xc000b24960)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/client-go@v0.0.0/util/workqueue/queue.go:200 +0x93
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).processNextWorkItem(0xc000b24b40)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apiserver@v0.0.0/pkg/server/dynamiccertificates/dynamic_cafile_content.go:225 +0x47
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).runWorker(...)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apiserver@v0.0.0/pkg/server/dynamiccertificates/dynamic_cafile_content.go:220
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:226 +0x33
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00007e9f0, {0x4b3ae00, 0xc0007a6f90}, 0x1, 0xc0004fc4e0)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:227 +0xaf
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00007e9f0, 0x3b9aca00, 0x0, 0x1, 0xc0004fc4e0)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:204 +0x7f
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery/pkg/util/wait.Until(...)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:161
12月 01 19:35:01 k8s-master-01 kubelet[857774]: created by k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).Run in goroutine 166
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apiserver@v0.0.0/pkg/server/dynamiccertificates/dynamic_cafile_content.go:161 +0x289
12月 01 19:35:01 k8s-master-01 kubelet[857774]: goroutine 147 [syscall]:
12月 01 19:35:01 k8s-master-01 kubelet[857774]: os/signal.signal_recv()
12月 01 19:35:01 k8s-master-01 kubelet[857774]: runtime/sigqueue.go:152 +0x29
12月 01 19:35:01 k8s-master-01 kubelet[857774]: os/signal.loop()
12月 01 19:35:01 k8s-master-01 kubelet[857774]: os/signal/signal_unix.go:23 +0x13
12月 01 19:35:01 k8s-master-01 kubelet[857774]: created by os/signal.Notify.func1.1 in goroutine 1
12月 01 19:35:01 k8s-master-01 kubelet[857774]: os/signal/signal.go:151 +0x1f
12月 01 19:35:01 k8s-master-01 kubelet[857774]: goroutine 142 [chan receive]:
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apiserver/pkg/server.SetupSignalContext.func1()
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apiserver@v0.0.0/pkg/server/signal.go:48 +0x25
12月 01 19:35:01 k8s-master-01 kubelet[857774]: created by k8s.io/apiserver/pkg/server.SetupSignalContext in goroutine 1
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apiserver@v0.0.0/pkg/server/signal.go:47 +0xda
12月 01 19:35:01 k8s-master-01 kubelet[857774]: goroutine 145 [select]:
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000750dc0, {0x4b3ae00, 0xc0007a6ea0}, 0x1, 0xc00010e720)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:238 +0x12c
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000750dc0, 0x2540be400, 0x0, 0x1, 0xc00010e720)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:204 +0x7f
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery/pkg/util/wait.Until(...)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:161
12月 01 19:35:01 k8s-master-01 kubelet[857774]: created by k8s.io/kubernetes/pkg/kubelet/certificate.addCertRotation in goroutine 1
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/kubernetes/pkg/kubelet/certificate/transport.go:156 +0x309
12月 01 19:35:01 k8s-master-01 kubelet[857774]: goroutine 162 [select]:
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/client-go/util/certificate.(*manager).Start.func1()
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/client-go@v0.0.0/util/certificate/certificate_manager.go:415 +0x212
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:226 +0x33
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000541ed8, {0x4b3ae00, 0xc0006b1740}, 0x1, 0xc0007fd920)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:227 +0xaf
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000541ed8, 0x3b9aca00, 0x0, 0x1, 0xc0007fd920)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:204 +0x7f
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery/pkg/util/wait.Until(...)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:161
12月 01 19:35:01 k8s-master-01 kubelet[857774]: created by k8s.io/client-go/util/certificate.(*manager).Start in goroutine 1
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/client-go@v0.0.0/util/certificate/certificate_manager.go:407 +0x167
12月 01 19:35:01 k8s-master-01 kubelet[857774]: goroutine 163 [chan receive]:
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc000b24960)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/client-go@v0.0.0/util/workqueue/queue.go:314 +0x9c
12月 01 19:35:01 k8s-master-01 kubelet[857774]: created by k8s.io/client-go/util/workqueue.newQueue in goroutine 1
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/client-go@v0.0.0/util/workqueue/queue.go:106 +0x1a5
12月 01 19:35:01 k8s-master-01 kubelet[857774]: goroutine 164 [select]:
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000b24ae0)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/client-go@v0.0.0/util/workqueue/delaying_queue.go:276 +0x2ff
12月 01 19:35:01 k8s-master-01 kubelet[857774]: created by k8s.io/client-go/util/workqueue.newDelayingQueue in goroutine 1
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/client-go@v0.0.0/util/workqueue/delaying_queue.go:113 +0x205
12月 01 19:35:01 k8s-master-01 kubelet[857774]: goroutine 165 [select]:
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/kubernetes/cmd/kubelet/app.BuildAuthn.func1.1()
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/kubernetes/cmd/kubelet/app/auth.go:103 +0x67
12月 01 19:35:01 k8s-master-01 kubelet[857774]: created by k8s.io/kubernetes/cmd/kubelet/app.BuildAuthn.func1 in goroutine 1
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/kubernetes/cmd/kubelet/app/auth.go:102 +0xac
12月 01 19:35:01 k8s-master-01 kubelet[857774]: goroutine 166 [chan receive]:
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).Run(0xc000b24b40, {0x4b6ace8, 0xc000751360}, 0x0?)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apiserver@v0.0.0/pkg/server/dynamiccertificates/dynamic_cafile_content.go:170 +0x375
12月 01 19:35:01 k8s-master-01 kubelet[857774]: created by k8s.io/kubernetes/cmd/kubelet/app.BuildAuthn.func1 in goroutine 1
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/kubernetes/cmd/kubelet/app/auth.go:110 +0x110
12月 01 19:35:01 k8s-master-01 kubelet[857774]: goroutine 167 [select]:
12月 01 19:35:01 k8s-master-01 kubelet[857774]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000b3f9a0, {0x4b6ace8, 0xc000751450})
12月 01 19:35:01 k8s-master-01 kubelet[857774]: google.golang.org/grpc@v1.56.3/internal/grpcsync/callback_serializer.go:83 +0x112
12月 01 19:35:01 k8s-master-01 kubelet[857774]: created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
12月 01 19:35:01 k8s-master-01 kubelet[857774]: google.golang.org/grpc@v1.56.3/internal/grpcsync/callback_serializer.go:55 +0x11a
12月 01 19:35:01 k8s-master-01 kubelet[857774]: goroutine 168 [select]:
12月 01 19:35:01 k8s-master-01 kubelet[857774]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000b3fa00, {0x4b6ace8, 0xc0007514f0})
12月 01 19:35:01 k8s-master-01 kubelet[857774]: google.golang.org/grpc@v1.56.3/internal/grpcsync/callback_serializer.go:83 +0x112
12月 01 19:35:01 k8s-master-01 kubelet[857774]: created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
12月 01 19:35:01 k8s-master-01 kubelet[857774]: google.golang.org/grpc@v1.56.3/internal/grpcsync/callback_serializer.go:55 +0x11a
12月 01 19:35:01 k8s-master-01 kubelet[857774]: goroutine 149 [select]:
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).watchCAFile(0xc000b24b40, 0xc0004fc4e0)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apiserver@v0.0.0/pkg/server/dynamiccertificates/dynamic_cafile_content.go:190 +0x2d5
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).Run.func1()
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apiserver@v0.0.0/pkg/server/dynamiccertificates/dynamic_cafile_content.go:165 +0x32
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:226 +0x33
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00052a7a0, {0x4b3ae00, 0xc0007a6f00}, 0x1, 0xc0004fc4e0)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:227 +0xaf
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00052a7a0, 0xdf8475800, 0x0, 0x1, 0xc0004fc4e0)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:204 +0x7f
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery/pkg/util/wait.Until(...)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apimachinery@v0.0.0/pkg/util/wait/backoff.go:161
12月 01 19:35:01 k8s-master-01 kubelet[857774]: created by k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).Run in goroutine 166
12月 01 19:35:01 k8s-master-01 kubelet[857774]: k8s.io/apiserver@v0.0.0/pkg/server/dynamiccertificates/dynamic_cafile_content.go:164 +0x358
12月 01 19:35:01 k8s-master-01 kubelet[857774]: goroutine 150 [IO wait]:
12月 01 19:35:01 k8s-master-01 kubelet[857774]: internal/poll.runtime_pollWait(0x7f9834c65e70, 0x72)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: runtime/netpoll.go:345 +0x85
12月 01 19:35:01 k8s-master-01 kubelet[857774]: internal/poll.(*pollDesc).wait(0xc00093e0c0?, 0xc000b8fe6b?, 0x1)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: internal/poll/fd_poll_runtime.go:84 +0x27
12月 01 19:35:01 k8s-master-01 kubelet[857774]: internal/poll.(*pollDesc).waitRead(...)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: internal/poll/fd_poll_runtime.go:89
12月 01 19:35:01 k8s-master-01 kubelet[857774]: internal/poll.(*FD).Read(0xc00093e0c0, {0xc000b8fe6b, 0x10000, 0x10000})
12月 01 19:35:01 k8s-master-01 kubelet[857774]: internal/poll/fd_unix.go:164 +0x27a
12月 01 19:35:01 k8s-master-01 kubelet[857774]: os.(*File).read(...)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: os/file_posix.go:29
12月 01 19:35:01 k8s-master-01 kubelet[857774]: os.(*File).Read(0xc000804058, {0xc000b8fe6b?, 0x0?, 0x0?})
12月 01 19:35:01 k8s-master-01 kubelet[857774]: os/file.go:118 +0x52
12月 01 19:35:01 k8s-master-01 kubelet[857774]: github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc000792af0)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: github.com/fsnotify/fsnotify@v1.6.0/backend_inotify.go:356 +0xcf
12月 01 19:35:01 k8s-master-01 kubelet[857774]: created by github.com/fsnotify/fsnotify.NewWatcher in goroutine 149
12月 01 19:35:01 k8s-master-01 kubelet[857774]: github.com/fsnotify/fsnotify@v1.6.0/backend_inotify.go:150 +0x186
12月 01 19:35:01 k8s-master-01 kubelet[857774]: goroutine 153 [runnable]:
12月 01 19:35:01 k8s-master-01 kubelet[857774]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000792cd0, 0x1)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: google.golang.org/grpc@v1.56.3/internal/transport/controlbuf.go:418 +0x113
12月 01 19:35:01 k8s-master-01 kubelet[857774]: google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0007b2620)
12月 01 19:35:01 k8s-master-01 kubelet[857774]: google.golang.org/grpc@v1.56.3/internal/transport/controlbuf.go:552 +0x86
12月 01 19:35:01 k8s-master-01 kubelet[857774]: google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
12月 01 19:35:01 k8s-master-01 kubelet[857774]: google.golang.org/grpc@v1.56.3/internal/transport/http2_client.go:451 +0x85
12月 01 19:35:01 k8s-master-01 kubelet[857774]: created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 169
12月 01 19:35:01 k8s-master-01 kubelet[857774]: google.golang.org/grpc@v1.56.3/internal/transport/http2_client.go:449 +0x22fb
12月 01 19:35:01 k8s-master-01 kubelet[857774]: runtime stack:
12月 01 19:35:01 k8s-master-01 kubelet[857774]: runtime.throw({0x4550fdf?, 0x0?})
12月 01 19:35:01 k8s-master-01 kubelet[857774]: runtime/panic.go:1023 +0x5c fp=0x7f9825ffac20 sp=0x7f9825ffabf0 pc=0x43fe9c
12月 01 19:35:01 k8s-master-01 kubelet[857774]: runtime.newstack()
12月 01 19:35:01 k8s-master-01 kubelet[857774]: runtime/stack.go:1047 +0x74d fp=0x7f9825ffadd0 sp=0x7f9825ffac20 pc=0x45c3cd
12月 01 19:35:01 k8s-master-01 kubelet[857774]: runtime.morestack()
12月 01 19:35:01 k8s-master-01 kubelet[857774]: runtime/asm_amd64.s:616 +0x7a fp=0x7f9825ffadd8 sp=0x7f9825ffadd0 pc=0x478f5a
12月 01 19:35:01 k8s-master-01 systemd[1]: kubelet.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
12月 01 19:35:01 k8s-master-01 systemd[1]: kubelet.service: Failed with result 'exit-code'.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
查看 /var/log/messages 日志
tail -100f /var/log/messages
1
Dec 1 20:00:44 k8s-master-01 kube-apiserver[866119]: #011k8s.io/kubernetes/cmd/kube-apiserver/apiserver.go:34 +0x18
Dec 1 20:00:44 k8s-master-01 systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Dec 1 20:00:44 k8s-master-01 systemd[1]: kube-apiserver.service: Failed with result 'exit-code'.
Dec 1 20:00:45 k8s-master-01 etcd[813]: {"level":"warn","ts":"2024-12-01T20:00:45.671837+0800","caller":"etcdserver/cluster_util.go:294","msg":"failed to reach the peer URL","address":"https://192.168.3.33:2380/version","remote-member-id":"8652603303dd8762","error":"Get \"https://192.168.3.33:2380/version\": dial tcp 192.168.3.33:2380: connect: connection refused"}
Dec 1 20:00:45 k8s-master-01 etcd[813]: {"level":"warn","ts":"2024-12-01T20:00:45.671876+0800","caller":"etcdserver/cluster_util.go:158","msg":"failed to get version","remote-member-id":"8652603303dd8762","error":"Get \"https://192.168.3.33:2380/version\": dial tcp 192.168.3.33:2380: connect: connection refused"}
Dec 1 20:00:46 k8s-master-01 etcd[813]: {"level":"warn","ts":"2024-12-01T20:00:46.012744+0800","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"8652603303dd8762","rtt":"0s","error":"dial tcp 192.168.3.33:2380: connect: connection refused"}
Dec 1 20:00:46 k8s-master-01 etcd[813]: {"level":"warn","ts":"2024-12-01T20:00:46.012803+0800","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"8652603303dd8762","rtt":"0s","error":"dial tcp 192.168.3.33:2380: connect: connection refused"}
1
2
3
4
5
6
7
2
3
4
5
6
7
192.168.3.33:2380 貌似有问题。查看该节点kube-apiserver 状态。systemctl status kube-apiserver
● kube-apiserver.service - Kubernetes API Server
Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; preset: disabled)
Active: active (running) since Sat 2024-11-30 13:09:03 CST; 1 day 6h ago
Docs: https://github.com/kubernetes/kubernetes
Main PID: 839 (kube-apiserver)
Tasks: 13 (limit: 23109)
Memory: 584.3M
CPU: 2h 10min 55.161s
CGroup: /system.slice/kube-apiserver.service
└─839 /usr/local/bin/kube-apiserver --v=2 --allow-privileged=true --bind-address=0.0.0.0 --secure-port=6443 --advertise-address=192.168.3.33 --service-cluster-ip-range=10.96.0.0/12,240e:378:17b:be10::69e/128 --service-node-port-range=30000-32767 --etcd-servers=https://192.168.3.31:2379,https://192.168.3.32:2379,https://192.168.3.33:2379 --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem --etcd-certfile=/etc/etcd/ssl/etcd.pem --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem --client-ca-file=/etc/kubernetes/pki/ca.pem --tls-cert-file=/etc/kubernetes/pki/apiserver.pem --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-account-signing-key-file=/etc/kubernetes/pki/sa.key --service-account-issuer=https://kubernetes.default.svc.cluster.local --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota --authorization-mode=Node,RBAC --enable-bootstrap-token-auth=true --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem --requestheader-allowed-names=aggregator --requestheader-group-headers=X-Remote-Group --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-username-headers=X-Remote-User --enable-aggregator-routing=true
12月 01 20:03:56 k8s-master-03 kube-apiserver[839]: , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
12月 01 20:03:56 k8s-master-03 kube-apiserver[839]: E1201 20:03:56.077779 839 available_controller.go:460] v3.projectcalico.org failed with: failing or missing response from https://172.31.44.60:5443/apis/projectcalico.org/v3: bad status from https://172.31.44.60:5443/apis/projectcalico.org/v3: 401
12月 01 20:03:57 k8s-master-03 kube-apiserver[839]: W1201 20:03:57.047451 839 handler_proxy.go:93] no RequestInfo found in the context
12月 01 20:03:57 k8s-master-03 kube-apiserver[839]: W1201 20:03:57.047465 839 handler_proxy.go:93] no RequestInfo found in the context
12月 01 20:03:57 k8s-master-03 kube-apiserver[839]: E1201 20:03:57.047509 839 controller.go:113] loading OpenAPI spec for "v3.projectcalico.org" failed with: Error, could not get list of group versions for APIService
12月 01 20:03:57 k8s-master-03 kube-apiserver[839]: I1201 20:03:57.047520 839 controller.go:126] OpenAPI AggregationController: action for item v3.projectcalico.org: Rate Limited Requeue.
12月 01 20:03:57 k8s-master-03 kube-apiserver[839]: E1201 20:03:57.047572 839 controller.go:102] loading OpenAPI spec for "v3.projectcalico.org" failed with: failed to download v3.projectcalico.org: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
12月 01 20:03:57 k8s-master-03 kube-apiserver[839]: , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
12月 01 20:03:57 k8s-master-03 kube-apiserver[839]: I1201 20:03:57.048788 839 controller.go:109] OpenAPI AggregationController: action for item v3.projectcalico.org: Rate Limited Requeue.
12月 01 20:03:58 k8s-master-03 kube-apiserver[839]: W1201 20:03:58.444953 839 logging.go:59] [core] [Channel #164 SubChannel #167] grpc: addrConn.createTransport failed to connect to {Addr: "192.168.3.33:2379", ServerName: "192.168.3.33", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 192.168.3.33:2379: connect: connection refused"
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
192.168.3.33:2379 有问题查看etcd 状态. systemctl status etcd
● etcd.service - Etcd Service
Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Sun 2024-12-01 20:06:39 CST; 8s ago
Docs: https://coreos.com/etcd/docs/latest/
Process: 805707 ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml (code=exited, status=1/FAILURE)
Main PID: 805707 (code=exited, status=1/FAILURE)
CPU: 46ms
1
2
3
4
5
6
7
2
3
4
5
6
7
继续 journalctl -u etcd -b 查看日志
12月 01 20:10:14 k8s-master-03 etcd[807226]: {"level":"info","ts":"2024-12-01T20:10:14.579015+0800","caller":"embed/etcd.go:136","msg":"configuring client listeners","listen-client-urls":["http://127.0.0.1:2379","https://192.168.3.33:2379"]}
12月 01 20:10:14 k8s-master-03 etcd[807226]: {"level":"warn","ts":"2024-12-01T20:10:14.579057+0800","caller":"embed/config.go:914","msg":"ignoring client auto TLS since certs given"}
12月 01 20:10:14 k8s-master-03 etcd[807226]: {"level":"info","ts":"2024-12-01T20:10:14.579090+0800","caller":"embed/etcd.go:620","msg":"pprof is enabled","path":"/debug/pprof"}
12月 01 20:10:14 k8s-master-03 etcd[807226]: {"level":"warn","ts":"2024-12-01T20:10:14.579118+0800","caller":"embed/etcd.go:627","msg":"scheme is http or unix while key and cert files are present; ignoring key and cert files","client-url":"http://127.0.0.1:2379"}
12月 01 20:10:14 k8s-master-03 etcd[807226]: {"level":"warn","ts":"2024-12-01T20:10:14.579144+0800","caller":"embed/etcd.go:630","msg":"scheme is http or unix while --client-cert-auth is enabled; ignoring client cert auth for this URL","client-url":"http://127.0.0.1:2379"}
12月 01 20:10:14 k8s-master-03 etcd[807226]: {"level":"info","ts":"2024-12-01T20:10:14.579310+0800","caller":"embed/etcd.go:310","msg":"starting an etcd server","etcd-version":"3.5.15","git-sha":"9a5533382","go-version":"go1.21.12","go-os":"linux","go-arch":"amd64","max-cpu-set":4,"max-cpu-available":4,"member-initialized":false,"name":"k8s-master-03","data-dir":"/var/lib/etcd","wal-dir":"/var/lib/etcd/wal","wal-dir-dedicated":"/var/lib/etcd/wal","member-dir":"/var/lib/etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":5000,"max-wals":5,"max-snapshots":3,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["https://192.168.3.33:2380"],"listen-peer-urls":["https://192.168.3.33:2380"],"advertise-client-urls":["https://192.168.3.33:2379"],"listen-client-urls":["http://127.0.0.1:2379","https://192.168.3.33:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"k8s-master-01=https://192.168.3.31:2380,k8s-master-02=https://192.168.3.32:2380,k8s-master-03=https://192.168.3.33:2380","initial-cluster-state":"new","initial-cluster-token":"etcd-k8s-cluster","quota-backend-bytes":2147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
12月 01 20:10:14 k8s-master-03 etcd[807226]: {"level":"info","ts":"2024-12-01T20:10:14.580510+0800","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/etcd/member/snap/db","took":"935.186µs"}
12月 01 20:10:14 k8s-master-03 etcd[807226]: {"level":"info","ts":"2024-12-01T20:10:14.600680+0800","caller":"embed/etcd.go:377","msg":"closing etcd server","name":"k8s-master-03","data-dir":"/var/lib/etcd","advertise-peer-urls":["https://192.168.3.33:2380"],"advertise-client-urls":["https://192.168.3.33:2379"]}
12月 01 20:10:14 k8s-master-03 etcd[807226]: {"level":"info","ts":"2024-12-01T20:10:14.601006+0800","caller":"embed/etcd.go:379","msg":"closed etcd server","name":"k8s-master-03","data-dir":"/var/lib/etcd","advertise-peer-urls":["https://192.168.3.33:2380"],"advertise-client-urls":["https://192.168.3.33:2379"]}
12月 01 20:10:14 k8s-master-03 etcd[807226]: {"level":"fatal","ts":"2024-12-01T20:10:14.601061+0800","caller":"etcdmain/etcd.go:204","msg":"discovery failed","error":"member 8652603303dd8762 has already been bootstrapped","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:204\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:31\nruntime.main\n\truntime/proc.go:267"}
12月 01 20:10:14 k8s-master-03 systemd[1]: etcd.service: Main process exited, code=exited, status=1/FAILURE
12月 01 20:10:14 k8s-master-03 systemd[1]: etcd.service: Failed with result 'exit-code'.
12月 01 20:10:14 k8s-master-03 systemd[1]: Failed to start Etcd Service.
1
2
3
4
5
6
7
8
9
10
11
12
13
2
3
4
5
6
7
8
9
10
11
12
13
# 问题的可能原因
- 重复引导问题：
- etcd 节点已经被初始化过，且数据目录中包含已有的元数据或快照。
- 尝试重新加入集群时，etcd 发现该节点的 UUID 已经存在。
- 数据目录未清理：
- 当重新配置 etcd 或改变集群拓扑时，没有清理
/var/lib/etcd数据目录。
- 当重新配置 etcd 或改变集群拓扑时，没有清理
- 配置冲突：
initial-cluster-state参数设置为new而不是existing，导致 etcd 认为这是一个全新的集群。initial-cluster参数与现有集群状态不匹配。
# 解决方案
根据当前集群状态修改配置文件 /etc/etcd/etcd.config.yml：
- 确保
initial-cluster-state设置为existing。 - 确保
initial-cluster参数与集群的实际拓扑一致。
重新启动 etcd：
systemctl start etcd
1