gitlabgitlab-ee

After upgrade to Gitlab-ee 17, there are problems


My box is Centos7, and by using 'yum install -y gitlab-ee' to upgrade gitlab-ee. After Gitlab-EE 17 upgraded, there are problems:

  1. Gitlab runner cannot be picked
  2. in Gitlab server, there is "Current Status: Unhealthy", via Admin Area -> Health Check
Migrations are pending. To resolve this issue, run: bin/rails db:migrate RAILS_ENV=production You have 103 pending migrations: 
20240205170838_change_approval_merge_request_rules_vulnerability_states_default.rb 
20240205171942_change_approval_project_rules_vulnerability_states_default.rb 
20240325131114_move_self_managed_cr_to_instance.rb 
20240328032449_drop_merge_request_diff_llm_summary_table.rb 
20240402143848_queue_backfill_jira_tracker_data_project_keys.rb 
20240403005214_add_concurrent_index_merge_requests_for_latest_diffs_with_state_merged.rb 
20240403005435_add_concurrent_index_on_merge_request_diffs_head_commit_sha.rb 
20240403104306_add_tmp_backfill_index_for_pipeline_ids_to_vulnerability_occurrences.rb 
20240408135326_remove_foreign_keys_from_external_approval_rules_protected_branches.rb 
20240408135652_drop_external_approval_rules_protected_branches_table.rb 
20240409070036_sync_index_for_pipelines_unique_id_bigint.rb 
20240410070036_prepare_async_indexes_for_pipelines_id_bigint.rb 
20240410104838_index_vulnerability_reads_on_state_report_type_severity_traversal_ids_archived.rb 
20240411070036_async_fk_id_bigint4_ci_pipelines_p_ci_builds_ci_pipelines_config_p_ci_stages.rb 
20240412000002_prepare_async_index_for_builds_part5.rb 
20240412125902_sync_index_for_p_ci_builds_part4.rb 
20240415122603_remove_inputs_from_catalog_resource_components.rb 
20240415130318_migrate_application_settings_help_text.rb 
20240416005004_swap_columns_for_p_ci_builds_runner_id.rb 
20240416094040_drop_ci_partition_sequence.rb 
20240416103114_ensure_backfill_packages_build_infos_pipeline_id_convert_to_bigint_is_completed.rb 
20240416103210_create_indexes_for_packages_build_infos_pipeline_id_convert_to_bigint.rb 
20240416110447_ensure_backfill_merge_trains_pipeline_id_convert_to_bigint_is_completed.rb 
20240416110522_create_indexes_for_merge_trains_pipeline_id_convert_to_bigint.rb 
20240416111927_ensure_backfill_vulnerability_feedback_id_convert_to_bigint_is_completed.rb 
20240416112009_create_index_for_vulnerability_feedback_pipeline_id_convert_to_bigint.rb 
20240416144510_migrate_container_protection_rules_minimum_access_level.rb 
20240416144520_cleanup_container_registry_protection_rule_protected_up_to_access_levels_rename.rb 
20240416144924_remove_catalog_resource_components_path_column.rb 
20240419035359_add_workspace_variables_project_id_trigger.rb 
20240419035360_queue_backfill_workspace_variables_project_id.rb 
20240419035507_add_work_item_parent_links_namespace_id_trigger.rb 
20240419035508_queue_backfill_work_item_parent_links_namespace_id.rb 
20240419035619_add_wiki_repository_states_project_id_trigger.rb 
20240419035620_queue_backfill_wiki_repository_states_project_id.rb 
20240419122328_swap_vulnerability_feedback_pipeline_id_convert_to_bigint.rb 
20240419131607_swap_packages_build_infos_pipeline_id_convert_to_bigint.rb 
20240421011547_sync_index_for_pipelines_id_bigint_related.rb 
20240421014253_validate_fk_id_bigint4_ci_pipelines_p_ci_builds_ci_pipelines_config_p_ci_stages.rb 
20240422070036_swap_ci_pipelines_pk_with_bigint_p_ci_builds_p_ci_stages.rb 
20240422080018_swap_merge_trains_pipeline_id_convert_to_bigint.rb 
20240422163959_queue_disable_allow_runner_registration_on_namespace_settings_for_gitlab_com.rb 
20240422164345_remove_user_details_onboarding_step_url_column.rb 
20240422164718_add_tmp_index_environments_on_flux_resource_path.rb 
20240422165035_update_kustomization_api_version.rb 
20240422165424_remove_tmp_index_environments_on_flux_resource_path.rb 
20240422232001_finalize_backfill_has_merge_request_of_vulnerability_reads.rb 
20240423020601_remove_idx_merge_requests_on_target_project_id_and_iid_opened.rb 
20240423022641_drop_idx_merge_requests_on_target_project_id_and_locked_state.rb 
20240423024034_drop_index_merge_requests_on_target_project_id_and_iid_and_state_id.rb 
20240423035245_sync_index_for_pipelines_id_bigint_part5.rb 
20240423035625_prepare_async_index_removal_for_vulnerabilities.rb 
20240423235307_swap_columns_for_p_ci_builds_project_id.rb 
20240424100836_ensure_backfill_merge_request_metrics_pipeline_id_convert_to_bigint_is_completed.rb 
20240424100929_create_indexes_for_merge_request_metrics_pipeline_id_convert_to_bigint.rb 
20240424103758_prepare_async_index_for_builds_part6.rb 
20240424111535_swap_merge_request_metrics_pipeline_id_convert_to_bigint.rb 
20240424120001_remove_unique_index_for_ml_model_packages_on_project_id_name_version.rb 
20240424120002_add_unique_index_for_ml_model_packages_on_project_id_name_version.rb 
20240424180330_remove_partition_p_ci_job_artifacts_project_id_idx.rb 
20240424183213_backfill_deployment_approval_data.rb 
20240425133709_finalize_feedback_to_state_transition_migration.rb 
20240425140717_finalize_vulnerability_links_creation.rb 
20240425170527_remove_foreign_keys_geo_event_log.rb 
20240425182054_remove_unused_columns_geo_event_log.rb 
20240425205205_queue_remove_namespace_from_os_type_sbom_components.rb 
20240426135340_prepare_async_index_to_execution_config_id_in_ci_build.rb 
20240429113537_ensure_backfill_vulnerability_occurrence_pipelines_id_to_bigint_is_completed.rb 
20240429113608_prepare_async_indexes_for_vulnerability_occurrence_pipelines_pipeline_to_bigint.rb 
20240429205901_remove_the_index_ci_pipeline_artifacts_on_pipeline_id.rb 
20240430004051_finalize_backfill_has_remediations_of_vulnerability_reads.rb 
20240430015450_sync_index_for_builds_user_id_bigint.rb 
20240430015514_swap_columns_for_p_ci_builds_user_id.rb 
20240430111455_finalize_backfill_vulnerability_reads_cluster_agent_migration.rb 
20240501044235_index_approval_merge_request_rule_sources_on_project_id.rb 
20240501044236_add_approval_merge_request_rule_sources_project_id_fk.rb 
20240501044237_add_approval_merge_request_rule_sources_project_id_trigger.rb 
20240501044238_queue_backfill_approval_merge_request_rule_sources_project_id.rb 
20240501201630_remove_code_suggestions_enabled_project_setting.rb 
20240502044605_remove_create_empty_embeddings_records_worker.rb 
20240502062514_add_foreign_key_from_pipeline_to_ci_builds_to_execution_configs.rb 
20240502120047_index_vulnerability_reads_for_common_group_level_query.rb 
20240503103337_queue_backfill_epic_basic_fields_to_work_item_record.rb 
20240503165628_remove_foreign_key_geo_hashed_storage_migrated_events.rb 
20240503170147_drop_table_geo_hashed_storage_migrated_events.rb 
20240503171707_remove_foreign_key_geo_hashed_storage_attachments_events.rb 
20240503171904_drop_table_geo_hashed_storage_attachements_events.rb 
20240503173034_remove_foreign_key_geo_repository_updated_events.rb 
20240503173210_drop_table_geo_repository_updated_events.rb 
20240503174054_remove_foreign_key_geo_repository_renamed_events.rb 
20240503174241_drop_table_geo_repository_renamed_events.rb 
20240503174832_remove_foreign_key_geo_repository_created_events.rb 
20240503175120_drop_table_geo_repository_created_events.rb 
20240503175735_drop_table_geo_repository_deleted_events.rb 
20240503180347_remove_foreign_key_geo_reset_checksum_events.rb 
20240503180517_drop_table_geo_reset_checksum_events.rb 
20240504042340_add_index_catalog_resources_on_usage_count.rb 
20240507194416_drop_index_abuse_reports_on_user_id.rb 
20240507194839_drop_index_board_group_recent_visits_on_user_id.rb 
20240507231644_add_index_members_on_lower_invite_email.rb 
20240508064453_drop_index_ci_pipeline_config_on_pipeline_id.rb 
20240508072011_drop_index_ci_runner_manager_build_on_runner_machine_id.rb 
20240508085441_re_add_redirect_routes_path_index.rb ..
  1. when run "gitlab-ctl check-config":
Malformed configuration JSON file found at /opt/gitlab/embedded/nodes/VM-12-10-centos.json.
This usually happens when your last run of `gitlab-ctl reconfigure` didn't complete successfully.
This file is used to check if any of the unsupported configurations are enabled,
and hence require a working reconfigure before upgrading.
Please run `sudo gitlab-ctl reconfigure` to fix it and try again.

4.when run "gitlab-ctl reconfigure":

Running handlers:
[2024-05-17T13:02:30+08:00] ERROR: Running exception handlers
There was an error running gitlab-ctl reconfigure:

version_file[Create version file for Gitlab KAS] (gitlab-kas::enable line 67) had an error: RuntimeError: Execution of the command `/opt/gitlab/embedded/bin/gitlab-kas --version` failed with a non-zero exit code (2)
stdout: 
stderr: panic: failed to parse "2024-05-15T09:09:49+0000" into RFC3339 compliant time object, because: parsing time "2024-05-15T09:09:49+0000" as "2006-01-02T15:04:05Z07:00": cannot parse "+0000" as "Z07:00". Fix the build process.

goroutine 1 [running]:
gitlab.com/gitlab-org/cluster-integration/gitlab-agent/v17/cmd.init.0()
    /var/cache/omnibus/src/gitlab-kas/cmd/build_info.go:18 +0xf3



Running handlers complete
[2024-05-17T13:02:30+08:00] ERROR: Exception handlers complete
Infra Phase failed. 2 resources updated in 14 seconds
[2024-05-17T13:02:30+08:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2024-05-17T13:02:30+08:00] FATAL: ---------------------------------------------------------------------------------------
[2024-05-17T13:02:30+08:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2024-05-17T13:02:30+08:00] FATAL: ---------------------------------------------------------------------------------------
[2024-05-17T13:02:30+08:00] FATAL: RuntimeError: version_file[Create version file for Gitlab KAS] (gitlab-kas::enable line 67) had an error: RuntimeError: Execution of the command `/opt/gitlab/embedded/bin/gitlab-kas --version` failed with a non-zero exit code (2)
stdout: 
stderr: panic: failed to parse "2024-05-15T09:09:49+0000" into RFC3339 compliant time object, because: parsing time "2024-05-15T09:09:49+0000" as "2006-01-02T15:04:05Z07:00": cannot parse "+0000" as "Z07:00". Fix the build process.

goroutine 1 [running]:
gitlab.com/gitlab-org/cluster-integration/gitlab-agent/v17/cmd.init.0()
    /var/cache/omnibus/src/gitlab-kas/cmd/build_info.go:18 +0xf3


Is there anyone facing this issue? Please help to fix, thanks very much.


Solution

  • If the kubernetes agent server gitlab_kas is not required you can try disabling it by adding the line gitlab_kas['enable'] = false to /etc/gitlab/gitlab.rb.

    After this, reattempting the gitlab-ctl reconfigure worked for us