We have around 5000 projects (various technologies) managed in StartTeam. Client moving to a new stack Bitbucket, JFrog, Bamboo & UrbanCode.
Bitbucket: It will be used for source control and underlying GIT will be used as revision control systems.
JFrog: Will manage the binaries for maven repository.
Bamboo: Will be used for build server.
UrbanCode: It will be used to automate the code deployment process.
I am taking JAVA project as reference for my questions. Currently StartTeam project contains source code as well as all required binaries and its not a maven project. ANT script is used for project build.
Requirement is migrating project like this to Bitbucket with minimal effort. the Bitbucket should not contain any binaries it would only manage the source code. Client also has setup an artifactory JFrog which will manage binaries for maven.
As part of this migration I am thinking a hybrid approach something like:
Step 1: Project will be downloaded from StartTeam.
Step 2: All binaries will be added to a new pom.xml as dependencies
Step 3: Code will be checkin to Bibucket
Step 4: In bamboo build server the build will be configured in two steps
a. First it will download all the required jars into a folder by executing the pom.xml
b. Then the existing ant script will be called to build the project by adding all the jars downloaded in previous step into CLASSPATH
Step5: UrbanCode will be configured to automate the deployment process
Already migrated few projects using this approach.
If time permits may be we will consider first fully mavenized the project (secondary approach) before importing into Bitbucket.
Questions:
1) There are approximately 5000 projects that need to be migrated so I am looking for expert suggestions how to proceed with approach 1 (hybrid approach)?
2) Please suggest is there any other approaches which can make this migration with less effort?
3) Any tools which can accelerate this migration?
Finally I have automated this migrations process as much as possible in the following way:
1) Download the project from StarTeam
2) I have Developed a JAVA utility which will scan the project workspace and dump all the jar details into an excel sheet. For each jar it will calculate the checksum (SHA-1) using the below code
public static String calculateChecksum(File javaJarFile) {
String filepath = javaJarFile.getAbsolutePath();
StringBuilder sb = new StringBuilder();
FileInputStream fis = null;
try {
MessageDigest md = MessageDigest.getInstance("SHA-1"); //40 character checksum
fis = new FileInputStream(filepath);
byte[] dataBytes = new byte[1024];
int nread = 0;
while ((nread = fis.read(dataBytes)) != -1)
md.update(dataBytes, 0, nread);
byte[] mdbytes = md.digest();
for (int i = 0; i < mdbytes.length; i++)
sb.append(Integer.toString((mdbytes[i] & 0xff) + 0x100, 16)
.substring(1));
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (fis != null)
fis.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return sb.toString();
}
Then it will query the Artifactory using the checksum value for the matching jar details like groupId, artifactId, version etc...If the jar details not found in Artifactory, then it will query MAVEN central repository for the matching jar details. Finally all the existing project jar details and the corresponding MAVEN compatible jar details will be dumped in the excel sheet.
Sample code for querying the MAVEN central repository
CloseableHttpClient httpClient = HttpClients.custom().build();
HttpPost getRequest = new HttpPost("http://search.maven.org/solrsearch/select?q=1:<JAR CHECKSUM>&rows=1&wt=json");
getRequest.addHeader("accept", "application/json");
HttpResponse response = httpClient.execute(getRequest);
if (response.getStatusLine().getStatusCode() != 200) {
throw new RuntimeException("Failed : HTTP error code : "
+ response.getStatusLine().getStatusCode());
}
BufferedReader br = new BufferedReader(new InputStreamReader(
(response.getEntity().getContent())));
String output;
StringBuffer outputBuffer = new StringBuffer("");
while ((output = br.readLine()) != null) {
outputBuffer.append(output);
}
JSONObject jsonObj = new JSONObject(outputBuffer.toString());
LOGGER.info("MAVEN Compatible Dependency Found: " + jsonObj.getJSONObject ("response").getInt("numFound"));
if (jsonObj.getJSONObject ("response").getInt("numFound") > 1) {
JSONArray jsonArray = jsonObj.getJSONObject ("response").getJSONArray("docs");
JSONObject object = (JSONObject) jsonArray.get(0);
LOGGER.info(object.getString("g"));
LOGGER.info(object.getString("a"));
LOGGER.info(object.getString("v"));
}
Sample code for querying the Artifactory
String checkSumUri = "https://anupg.jfrog.io/anupg/api/search/checksum?sha1=" + checkSum;
HttpResponse rtFactResponse = callService (checkSumUri, true);
BufferedReader br = new BufferedReader(new InputStreamReader((rtFactResponse.getEntity().getContent())));
String output;
StringBuffer outputBuffer = new StringBuffer("");
while ((output = br.readLine()) != null) {
outputBuffer.append(output);
}
String uri = null;
if (!outputBuffer.toString().trim().equals("")) {
JSONObject jsonObj = new JSONObject(outputBuffer.toString());
JSONArray jsonArray = jsonObj.getJSONArray("results");
for (int i=0; i < jsonArray.length(); i++) {
JSONObject jsonUriObject = (JSONObject) jsonArray.get(i);
System.out.println("URI---------->" + jsonUriObject.getString("uri").replace(".jar", ".pom"));
uri = jsonUriObject.getString("uri").replace(".jar", ".pom");
}
}
if (uri != null) {
String downloadURI = null;
HttpResponse uriResponse = callService (uri);
BufferedReader uriBR = new BufferedReader(new InputStreamReader(
(uriResponse.getEntity().getContent())));
String uriOutput;
StringBuffer uriOutputBuffer = new StringBuffer("");
while ((uriOutput = uriBR.readLine()) != null) {
uriOutputBuffer.append(uriOutput);
}
if (!uriOutputBuffer.toString().trim().equals("")) {
JSONObject jsonUriObject = new JSONObject(uriOutputBuffer.toString());
System.out.println("Download URI---------->" + jsonUriObject.getString("downloadUri"));
downloadURI = jsonUriObject.getString("downloadUri");
}
HttpResponse downloadUriResponse = callService (downloadURI, true);
if (downloadUriResponse.getStatusLine().getStatusCode() != 200) {
throw new RuntimeException("Failed : HTTP error code : "
+ downloadUriResponse.getStatusLine().getStatusCode());
}
InputStream is = downloadUriResponse.getEntity().getContent();
DocumentBuilderFactory dbFactory = DocumentBuilderFactory.newInstance();
Document doc = null;
try {
DocumentBuilder dBuilder = dbFactory.newDocumentBuilder();
doc = dBuilder.parse(is);
doc.getDocumentElement().normalize();
} catch (ParserConfigurationException e) {
e.printStackTrace();
} catch (SAXException e) {
e.printStackTrace();
} finally {
if (is != null) {
is.close();
}
}
System.out.println("root of xml file: " + doc.getDocumentElement().getNodeName());
Element element = doc.getDocumentElement();
if (getNodeValue("groupId", element) != null &&
getNodeValue("artifactId", element) != null &&
getNodeValue("version", element) != null) {
metadata.setGroupId(getNodeValue("groupId", element));
metadata.setArtifactId(getNodeValue("artifactId", element));
metadata.setVersion(getNodeValue("version", element));
metadata.setRTfactoryFound(true);
}
}
There could be a chance that project may contain custom jars which may not be available either in Artifactory or MAVEN central. In such cases the generated excel sheet will be shared with AD team for corresponding jar details. The jar will be installed in Artifactory and the AD team will update the excel sheet with the groupId, arfactId, version.
Once the excel sheet having all the jar details the JAVA utility will generate the pom.xml by reading the corresponding MAVEN details. The "copy-rename-maven-plugin" plugin is used here to copy all the maven downloaded jars from "target/dependency" folder to respective project folder. This pom will be configured as the first build steps in Bamboo server followed by build.xml which will build the project. Note I am using hybrid approach in build process.
Below is the code snippets for the same
Writer w = null;
MavenXpp3Writer mavenXpp3Writer = new MavenXpp3Writer();
try {
List<JarDetails> uniquejarDetails = removeDuplicateJars (jarDetails);
w = WriterFactory.newWriter (new File(location + "pom.xml"), "UTF-8");
Model model = new Model();
model.setGroupId("com.projectname");
model.setArtifactId("project-analyzer");
model.setVersion("1.0");
model.setModelVersion("4.0.0");
List<Dependency> dependencies = new ArrayList<Dependency>();
Plugin copyDependency = new Plugin();
copyDependency.setGroupId("org.apache.maven.plugins");
copyDependency.setArtifactId("maven-dependency-plugin");
copyDependency.setVersion("2.10");
PluginExecution copyDependencyPluginExecution = new PluginExecution();
copyDependencyPluginExecution.setId("copy-dependencies");
copyDependencyPluginExecution.setPhase("generate-sources");
List<String> copyDependencyGoalsList = new ArrayList<String>();
copyDependencyGoalsList.add("copy-dependencies");
copyDependencyPluginExecution.setGoals(copyDependencyGoalsList);
Plugin plugin = new Plugin();
plugin.setGroupId("com.coderplus.maven.plugins");
plugin.setArtifactId("copy-rename-maven-plugin");
plugin.setVersion("1.0.1");
PluginExecution pluginExecution = new PluginExecution();
pluginExecution.setId("copy-jars");
pluginExecution.setPhase("generate-sources");
List<String> goalsList = new ArrayList<String>();
goalsList.add("copy");
pluginExecution.setGoals(goalsList);
String domString = "<configuration><fileSets>";
for (int jarCount = 0; jarCount < uniquejarDetails.size(); jarCount++) {
if (uniquejarDetails.get(jarCount).getDependencyMetadata().getGroupId() != null) {
Dependency dependency = new Dependency();
dependency.setGroupId(uniquejarDetails.get(jarCount).getDependencyMetadata().getGroupId());
dependency.setArtifactId(uniquejarDetails.get(jarCount).getDependencyMetadata().getArtifactId());
dependency.setVersion(uniquejarDetails.get(jarCount).getDependencyMetadata().getVersion());
dependencies.add(dependency);
//Add copy-rename-maven-plugin configurations
String mavenJarName = uniquejarDetails.get(jarCount).getDependencyMetadata().getArtifactId() + "-"
+ uniquejarDetails.get(jarCount).getDependencyMetadata().getVersion() + ".jar";
String mavenJar = "target/dependency/" + mavenJarName;
domString += "<fileSet><sourceFile>" + mavenJar + "</sourceFile>";
domString += "<destinationFile>" + new File(uniquejarDetails.get(jarCount).getJarRelativePath() +
"/" + mavenJarName) + "</destinationFile></fileSet>";
}
}
domString += "</fileSets></configuration>";
InputSource is = new InputSource();
is.setCharacterStream(new StringReader(domString));
Xpp3Dom dom = Xpp3DomBuilder.build(new StringReader(domString));
pluginExecution.setConfiguration(dom);
List<PluginExecution> pluginExecuionList = new ArrayList<PluginExecution>();
pluginExecuionList.add(pluginExecution);
List<PluginExecution> copyDependencyPluginExecuionList = new ArrayList<PluginExecution>();
copyDependencyPluginExecuionList.add(copyDependencyPluginExecution);
plugin.setExecutions (pluginExecuionList);
copyDependency.setExecutions (copyDependencyPluginExecuionList);
List<Plugin> pluginList = new ArrayList<Plugin> ();
pluginList.add(copyDependency);
pluginList.add(plugin);
Build build = new Build();
build.setPlugins(pluginList);
model.setDependencies(dependencies);
model.setBuild(build);
mavenXpp3Writer.write(w, model);
} catch (UnsupportedEncodingException e) {
LOGGER.error(e.getMessage(), e);
} catch (FileNotFoundException e) {
LOGGER.error(e.getMessage(), e);
} catch (IOException e) {
LOGGER.error(e.getMessage(), e);
} catch (FactoryConfigurationError e) {
e.printStackTrace();
} catch (XmlPullParserException e) {
e.printStackTrace();
} finally {
try {
if (w != null)
w.close();
} catch (IOException e) {
LOGGER.error (e.getMessage(), e);
}
}
Next steps is cleanup: It will delete all the jars from the project and will keep a track of the jar locations.
Created a shell script which will connect the bitbucket server and will push the code using git command. I am using "GIT Bash" to execute the script. Note, I am using Bitbucket rest API to create the project and repositories remotely before pushing the code.
REST service details for creating the project:
curl -X POST -v -u $bitbucket_user:${bitbucket_password} -H "Content-Type: application/json" "http://hostname:7990/rest/api/1.0/projects" -d "{\"key\": \"$project_key\",\"name\": \"$project_name\", \"description\": \"$project_desc\"}" > response.json
REST service details for creating repository under the above Project
curl -X POST -v -u $bitbucket_user:${bitbucket_password} -H "Content-Type: application/json" "http://hostname:7990/rest/api/1.0/projects/$project_key/repos" -d "{\"name\": \"$repository_name\", \"scmId\": \"git\", \"forkable\":true}" > repo-response.json
Git commands for pushing the project into Bitbucket
git init
git add .
git commit -m "Initial commit"
git remote add origin http://$bitbucket_user:${bitbucket_password}@hostname:7990/scm/${project_key}/${repository_name}.git
git push -u origin --all
Using this approach we have automated 80% of the migration activity.