Managing GKE (k8s) clusters and applications inside them has become a never ending battle. Managing apps and configuration accurately has become cumbersome. This has become more common given the rise of micro-services and event-driven services.
With cloud providing unlimited resources, we may end up with a huge bill if not careful. A good approach is to start very small and expand slowly based on metrics rather than upfront provisioning. You can use tools like terraform, GKEs, grafana for this purpose.