Using you own Maven repository with Aspire

From wiki.searchtechnologies.com
Jump to: navigation, search

For Information on Aspire 3.1 Click Here

Introduction

Consider the following scenario - you wish to run Aspire, making use of the Aspire features that allow applications and components to be fetched from the internet automatically, but your corporate network policy does not allow servers to have uncontrolled access to the internet. You anticipate a number of different Aspire installations and don't want the overhead of adding a new exception to the firewall everytime you want to deploy Aspire to a new server. You may also be considering developing your own Aspire applications or components and be looking for somewhere to deploy them to so they can be easily used on all your Aspire installations.

Setting up your own Maven repository could be the solution.

On this page we describe how to connect an Artifactory instance (see http://http://www.jfrog.com - either the open source or Pro version will do) behind a firewall to Search Technologies' Artifactory repository and the changes required to the Maven and Aspire settings files in order to acheive this.

Solution Overview

Our solution will involve installing an Artifactory instance on a server inside the corporate firewall and allowing this access to the internet. Any requests from Maven or Aspire will then be sent to this internal instance and that will serve the responses, accessing external repositories as appropriate. Downloaded artifacts will be cached on the internal instance, giving the added benefit of reduced external traffic.

ArtifactoryArchitectureOverview.jpg

This page documents a solution based on Linux servers, but should be equally applicable to a Windows environment.

Artifactory server

We will install and configure an Artifactory server and give this server access to specific Maven repositories on the internet. All requests for Maven artifacts from inside the firewall will be sent to this server which will attempt to satisfy them from its internal repositories, or from caches of external or remote repositories and finally from the external or remote repositories themselves.

We'll assume the following in this section:

  • Artifactory will be installed on server artifactory.myCompany.com on the default port (8081).

You'll need the email address and password that you used when registering for Aspire.

The entire Artifactory documentation is available here.

Network access requirements

This server will need network access to any remote repository registered in Artifactory. Specifically, it will need access to the Search Technologies' repository (http://repository.searchtechnologies.com/artifactory/simple/community-public/). The default installation of Artifactory registers a number of remote repositories, so including the Search Technologies' repository, the server will need network access to:

  • repository.searchtechnologies.com
  • repo1.maven.org:80
  • download.java.net:80
  • repo.jfrog.org:80
  • repository.jboss.org:80
  • repository.codehaus.org:80
  • maven.springframework.org:80
  • google-maven-repository.googlecode.com:80
  • gradle.artifactoryonline.com:80

In order to allow configuration of this Artifactory instance, the server will need to accept incoming traffic on port 80.

Downloading and installing Artifactory

Download and install Artifactory following the instructions from here.

Adding the Search Technologies repository

Log in to the internal Artifactory instance by entering http://artifactory.myCompany.com:8081/artifactory (default admin/password)

Remote repository (Basic settings)

Using the admin tab, navigate to Configuration/Repositories on the left hand side menu and add a new Remote Repository. On the Basic Settings tab add the Repository Key (SearchTechnologies) and URL (http://repository.searchtechnologies.com/artifactory/simple/community-public/).

Remote repository (Advanced settings)

On the AdvancedSettings tab, under Network, add the User (email address) and Password you used when registering Aspire. If you want to be able to browse the repository, scroll to the bottom and check the Allow Archive Browsing checkbox.

Use the Test button to test your connection to the Search Technologies' repository.

NOTE: Adding a new Remote repository does not automatically add it to the Virtual repository of all remote repositories so you will ned to add it manually by editing the Virtual repository remote-repos at the bottom of the Configuration/Repositories page.

Adding a local repository

If you want to add a local repository, giving you somewhere to deploy your bespoke Aspire applications and components, log in to the internal Artifactory instance by entering http://artifactory.myCompany.com:8081/artifactory (default admin/password)

Local repository (Basic settings)

Using the admin tab, navigate to Configuration/Repositories on the left hand side menu and add a new Local Repository. On the Basic Settings tab add the Repository Key (enter any name you like). If you want to be able to browse the repository, scroll to the bottom and check the Allow Archive Browsing checkbox.

Adding security

Typically you'll want to add some security too. The minimum is likely to consist of a permission set that grants access to the internal repository (myCompany-internal from above) and any remote repository (which will include the Search Technologies' repository), a group that gives access to the permission set and a user that belongs to that group.

See here for details.

Aspire server

This section describes the additional configuration required to use the new internal repository you created above.

We'll assume the following in this section:

  • The internal repository is hosted on the server artifactory.myCompany.com:8081
  • The internal repository is named myCompany-internal
  • The remote (Search Technologies) repository is named SearchTechnologies
  • There is a virtual repository (referencing all remote repositories) named remote-repos
  • The user name and password to access the local Artifactory instance is myLocalUser and myLocalPassword. This user will have read access to the local and remote repositories.

In order to build Aspire distributions on this server, we'll configure Maven to use the internal repository and to use a mirror for all other requests.

Network access requirements

In order to access Maven plugins and Aspire applications and components, this server will not need to access the internet and will only need access to the internal Artifactory instance on artifactory.myCompany.com:8081 and will . However, in order to allow incoming connections to the Aspire interfaces, access to this server on port 50505 will also need to be allowed.

Downloading and installing the Maven client

Download the Maven command line client as per the instructions at http://maven.apache.org/download.cgi.

The Maven settings file

The Maven settings file is used when we download the Aspire distributions, or when we build Aspire. We need to configure the Maven settings file for access to the remote repositories and the local repository (if there is one). We want to be able to send requests for artifacts from the local repository direct, but have any other requests sent to the local instance and satisfied from the caches there or then sent out to the internet (by the local instance).

The settings file is located in $M2_HOME/conf. To access the local repository, we need a maven settings file that looks like this:

 <?xml version="1.0" encoding="UTF-8"?>
 <settings>
   <servers>
     <server>
       <id>myCompanyServer</id>
       <username>myLocalUser</username>
       <password>myLocalPassword</password>
     </server>
   </servers>
   <profiles>
     <profile>
       <id>myCompanyRepo</id>
       <repositories>
         <repository>
           <id>myCompanyServer</id>
           <url>http://artifactory.myCompany.com:8081/artifactory/simple/myCompany-internal/</url>
           <snapshots>
             <enabled>true</enabled>
             <updatePolicy>always</updatePolicy>
           </snapshots>
           <releases>
             <enabled>true</enabled>
             <updatePolicy>always</updatePolicy>
           </releases>
         </repository>
       </repositories>
     </profile>
   </profiles>
   <activeProfiles>
     <activeProfile>myCompanyRepo</activeProfile>
   </activeProfiles>
 </settings>

This is basically the Maven settings file documented here but with the ids and urls changed.

However, this will not cause Maven to use the local instance to resolve requests for remote repositories. To do this, we need to add a mirrors section:

 <?xml version="1.0" encoding="UTF-8"?>
 <settings>
   <servers>
     <server>
       <id>myCompanyServer</id>
       <username>myLocalUser</username>
       <password>myLocalPassword</password>
     </server>
   </servers>
   <mirrors>
     <mirror>
       <mirrorOf>*</mirrorOf>
       <name>remote-repos</name>
       <url>http://artifactory.myCompany.com:8081/artifactory/remote-repos</url>
       <id>myCompanyServer</id>
     </mirror>
   </mirrors>
   <profiles>
     <profile>
       <id>myCompanyRepo</id>
       <repositories>
         <repository>
           <id>myCompanyServer</id>
           <url>http://artifactory.myCompany.com:8081/artifactory/simple/myCompany-internal/</url>
           <snapshots>
             <enabled>true</enabled>
             <updatePolicy>always</updatePolicy>
           </snapshots>
           <releases>
             <enabled>true</enabled>
             <updatePolicy>always</updatePolicy>
           </releases>
         </repository>
       </repositories>
     </profile>
   </profiles>
   <activeProfiles>
     <activeProfile>myCompanyRepo</activeProfile>
   </activeProfiles>
 </settings>

This will cause all requests for repositories other than the local instance to be served from the local instance.

Creating an Aspire distribution

Once Maven has been configured, you can create an Aspire distribution using the Archetype (see here and here).

Use the command:

 mvn archetype:generate -DarchetypeGroupId=com.searchtechnologies -DarchetypeArtifactId=aspire-distribution-archetype -DarchetypeVersion=1.1
   -DarchetypeRepository=http://artifactory.myCompany.com:8081/artifactory/public/SearchTechnologies

NOTE: This is slightly different to the command, noted on the referenced pages above, for use when not behind a firewall:

If you do not specify the -DarchetypeRepository parameter, the archetype plugin will attempt to connect directly to the Maven central repository (http://repo1.maven.org) but will not be able to (since it is behind the firewall) and will see an error similar to the following:

[WARNING] Error reading archetype catalog http://repo1.maven.org/maven2
org.apache.maven.wagon.TransferFailedException: Connection to http://repo1.maven.org refused
        at org.apache.maven.wagon.shared.http4.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:799)
        at org.apache.maven.wagon.StreamWagon.getInputStream(StreamWagon.java:116)
        at org.apache.maven.wagon.StreamWagon.getIfNewer(StreamWagon.java:88)
        at org.apache.maven.wagon.StreamWagon.get(StreamWagon.java:61)
        at org.apache.maven.archetype.source.RemoteCatalogArchetypeDataSource.downloadCatalog(RemoteCatalogArchetypeDataSource.java:119)
        at org.apache.maven.archetype.source.RemoteCatalogArchetypeDataSource.getArchetypeCatalog(RemoteCatalogArchetypeDataSource.java:87)
        at org.apache.maven.archetype.DefaultArchetypeManager.getRemoteCatalog(DefaultArchetypeManager.java:216)
        at org.apache.maven.archetype.DefaultArchetypeManager.getRemoteCatalog(DefaultArchetypeManager.java:205)
        at org.apache.maven.archetype.ui.generation.DefaultArchetypeSelector.getArchetypesByCatalog(DefaultArchetypeSelector.java:200)
        at org.apache.maven.archetype.ui.generation.DefaultArchetypeSelector.selectArchetype(DefaultArchetypeSelector.java:71)
        at org.apache.maven.archetype.mojos.CreateProjectFromArchetypeMojo.execute(CreateProjectFromArchetypeMojo.java:197)
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
        at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
        at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
Caused by: org.apache.maven.wagon.providers.http.httpclient.conn.HttpHostConnectException: Connection to http://repo1.maven.org refused
        at org.apache.maven.wagon.providers.http.httpclient.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:158)
        at org.apache.maven.wagon.providers.http.httpclient.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:149)
        at org.apache.maven.wagon.providers.http.httpclient.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121)
        at org.apache.maven.wagon.providers.http.httpclient.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:573)
        at org.apache.maven.wagon.providers.http.httpclient.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425)
        at org.apache.maven.wagon.providers.http.httpclient.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
        at org.apache.maven.wagon.providers.http.httpclient.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
        at org.apache.maven.wagon.shared.http4.AbstractHttpClientWagon.execute(AbstractHttpClientWagon.java:674)
        at org.apache.maven.wagon.shared.http4.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:793)
        ... 31 more
Caused by: java.net.ConnectException: Connection timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
        at java.net.Socket.connect(Socket.java:546)
        at org.apache.maven.wagon.providers.http.httpclient.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:123)
        at org.apache.maven.wagon.providers.http.httpclient.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148)
        ... 39 more
[WARNING] No archetype found in remote catalog. Defaulting to internal catalog
[WARNING] Archetype not found in any catalog. Falling back to central repository (http://repo1.maven.org/maven2).
[WARNING] Use -DarchetypeRepository=<your repository> if archetype's repository is elsewhere.

As the distribution is generated, it asks for your Maven user and password. You can use any value you like here - we'll need to change the settings file these are written to anyway (see below).

The Aspire settings file

The Aspire settings file also needs to be modified to use the internal instance of Artifactory. When you generate a distribution, you're asked for the Maven user and password values and these are written to the Aspire settings file. Unfortunately, the repository URL is hardcoded in the file, so we need to manually change it to point it to the cached copy on the local instance.

To do this, edit the distribution-files/config/settings.xml file and change the repositories section to look like this:

  <!-- Configured Repositories -->
  <repositories>
    <repository type="distribution">
      <directory>bundles/aspire</directory>
    </repository>

    <repository type="maven">
      <defaultVersion>1.1</defaultVersion>
      <remoteRepositories>
        <remoteRepository>
          <id>SearchTech</id>
          <url>http://artifactory.myCompany.com:8081/artifactory/SearchTechnologies/</url>
          <user>myMavenUser</user>
          <password>myMavenPassword</password>
        </remoteRepository>
    </repository>
  </repositories>

This points Aspire at the cache of the Search Technologies' repository on the local instance.

If you want to add applications or components from your local repository (myCompany-internal), you'll also need to add that to the Aspire configuration:

  <!-- Configured Repositories -->
  <repositories>
    <repository type="distribution">
      <directory>bundles/aspire</directory>
    </repository>

    <repository type="maven">
      <defaultVersion>1.1</defaultVersion>
    <repository type="maven">
      <defaultVersion>1.1</defaultVersion>
      <remoteRepositories>
        <remoteRepository>
          <id>SearchTech</id>
          <url>http://artifactory.myCompany.com:8081/artifactory/SearchTechnologies/</url>
          <user>myMavenUser</user>
          <password>myMavenPassword</password>
        </remoteRepository>
        <remoteRepository>
          <id>myCompanyInternal</id>
          <url>http://artifactory.myCompany.com:8081/artifactory/simple/myCompany-internal/</url>
          <user>myMavenUser</user>
          <password>myMavenPassword</password>
        </remoteRepository>
      </remoteRepositories>
    </repository>

  </repositories>

NOTE: both <remoteRepository> tags have the same <user> and <password>. This is because this is the password for the local instance - passwords for any remote repositories will have been configured in the repository configuration in the local instance.

Other Configurations

Using Artifactory there are a number of different configurations that could achieve the same over all effect. Artifactory implements virtual repositories where you can assign a name to a collection of repositories (both internal or external). You could therefore create a virtual repository made up of all the remote and internal repositories and then only need to configure a single repository in both the Maven and Aspire settings files. However, you cannot deploy to virtual repositories.