Difference between revisions of "RSS Connector App-bundle (Aspire 2)"

From wiki.searchtechnologies.com
Jump to: navigation, search

For Information on Aspire 3.1 Click Here

 
Line 6: Line 6:
 
|appBundleName=RSS Connector
 
|appBundleName=RSS Connector
 
|mavenCoordinates=com.searchtechnologies.aspire:app-rss-connector
 
|mavenCoordinates=com.searchtechnologies.aspire:app-rss-connector
|versions=2.0
+
|versions={{CurrentVersion2.X}}
 
|typeFlags=scheduled
 
|typeFlags=scheduled
 
}}  
 
}}  

Latest revision as of 00:12, 9 December 2015


RSS Connector App-bundle (Aspire 2)
AppBundle Name  RSS Connector
Maven Coordinates  com.searchtechnologies.aspire:app-rss-connector
Versions  2.2.2
Type Flags  scheduled
Inputs  AspireObject from a content source submitter holding all the information required for a crawl.
Outputs  An AspireObject containing the URL, content, ACLs and Metadata processed for each file.

The RSS Connector crawls content from RSS feeds and will extract metadata and content from each file downloaded. You can configure one or more RSS feeds to scan. Each item from the feed will be downloaded and will be tagged as and added document and can be routed to any Aspire pipeline as desired.

The connector, once started, can be stopped, paused or resumed sending a new Scanner Configuration Job. Typically the start job will contain all information required by the job to perform the scan. When pausing or stopping, the connector will wait until all the jobs it published have completed before updating the statistics and status of the connector.

NOTE that the RSS connector does not support document level security, and any jobs published will contain no access control lists (ACLs) for the scanned content.


Configuration

This section lists all configuration parameters available to install the RSS Connector Application Bundle and to execute crawls using the connector.

General Application Configuration

Property Type Default Description
snapshotDir string ${aspire.home}/snapshots The directory for snapshot files to be stored.
disableTextExtract boolean false By default, connectors use Apache Tika to extract text from downloaded documents. If you wish to apply special text processing to the downloaded document in the workflow, you should disable text extraction. The downloaded document is then available as a content stream.
workflowReloadPeriod int 15m The period after which to reload the business rules. Defaults to ms, but can be suffixed with ms, s, m, h or d to indicate the required units.
workflowErrorTolerant boolean false When set, exceptions in workflow rules will only effect the execution of the rule in which the exception occurs. Subsequent rules will be executed and the job will complete the workflow sucessfully. If not set, exceptions in workflow rules will be re-thrown and the job will be moved to the error workflow.
debug Boolean false Controls whether debugging is enabled for the application. Debug messages will be written to the log files.


FTP Connector Specific Configuration

There are no configuration options specific to the RSS connector

Configuration Example

To install the application bundle, add the configuration, as follows, to the <autoStart> section of the Aspire settings.xml.

<application config="com.searchtechnologies.aspire:app-rss-connector">
  <properties>
    <property name="generalConfiguration">true</property>
    <property name="snapshotDir">${dist.data.dir}/${app.name}/snapshots</property>
    <property name="workflowReloadPeriod">15s</property>
    <property name="batchSize">50</property>
    <property name="batchTimeout">60000</property>
    <property name="waitForSubJobs">600000</property>
    <property name="maxThreads">10</property>
    <property name="jobQueue">30</property>
    <property name="extractTimeout">180000</property>
    <property name="extractTextMaxSize">unlimited</property>
    <property name="disableTextExtract">false</property>
    <property name="workflowErrorTolerant">false</property>
    <property name="emitStartJob">true</property>
    <property name="emitEndJob">true</property>
    <property name="enableAuditing">true</property>
    <property name="debug">false</property>
    <property name="non-text-document">true</property>
    <property name="nonTextDocumentsExtensions">jpg,gif,mp3,mp4,mpg,avi,wav,bmp,swf</property>
    <property name="enableFetchUrl">true</property>
    <property name="fullRecovery">incremental</property>
    <property name="incrementalRecovery">incremental</property>
    <property name="fdService">false</property>
    <property name="fdServiceUrl"/>
  </properties>
</application>

Note: Any optional properties can be removed from the configuration to use the default value described on the table above.

Source Configuration

Scanner Control Configuration

The following table describes the list of attributes that the AspireObject of the incoming scanner job requires to correctly execute and control the flow of a scan process.

Element Type Options Description
@action string start, stop, pause, resume, abort Control command to tell the scanner which operation to perform. Use start option to launch a new crawl.
@actionProperties string full, incremental When a start @action is received, it will tell the scanner to either run a full or an incremental crawl.
@normalizedCSName string Unique identifier name for the content source that will be crawled.
displayName string Display or friendly name for the content source that will be crawled.

Header Example

  <doc action="start" actionProperties="full" actionType="manual" crawlId="0" dbId="0" jobNumber="0" normalizedCSName="FeedOne_Connector"
   scheduleId="0" scheduler="##AspireSystemScheduler##" sourceName="ContentSourceName">
    ...
    <displayName>testSource</displayName>
    ...
  </doc>

All configuration properties described in this section are relative to /doc/connectorSource of the AspireObject of the incoming Job.

Property Type Default Description
feedUrls/feedUrl (Multiple allowed) string The URL of the RSS feed to crawl
indexContainers boolean false true if folders (as well as files) should be indexed.

Scanner Configuration Example

  <doc action="start" actionProperties="full" actionType="manual" normalizedCSName="FTP_Connector" sourceName="FTP_Connector">
    <connectorSource>
      <feedUrls>
        <feedUrl>http://feeds.bbci.co.uk/news/rss.xml?edition=uk</feedUrl>
        <feedUrl>http://feeds.skynews.com/feeds/rss/world.xml</feedUrl>
        <feedUrl>http://feeds.skynews.com/feeds/rss/us.xml</feedUrl>
        <feedUrl>http://www.telegraph.co.uk/news/uknews/rss</feedUrl>
        <feedUrl>http://www.telegraph.co.uk/news/rss</feedUrl>
        <feedUrl>http://www.telegraph.co.uk/sport/rss</feedUrl>
      </feedUrls>
      <indexContainers>false</indexContainers>
    </connectorSource>
    <displayName>FTP_Connector</displayName>
  </doc>

Example Output

<doc>
  <url>http://www.bbc.co.uk/news/world-europe-33284937#sa-ns_mchannel=rss&ns_source=PublicRSS20-sa</url>
  <id>http://www.bbc.co.uk/news/world-europe-33284937#sa-ns_mchannel=rss&ns_source=PublicRSS20-sa</id>
  <fetchUrl>http://www.bbc.co.uk/news/world-europe-33284937#sa-ns_mchannel=rss&ns_source=PublicRSS20-sa</fetchUrl>
  <displayUrl>http://www.bbc.co.uk/news/world-europe-33284937#sa-ns_mchannel=rss&ns_source=PublicRSS20-sa</displayUrl>
  <snapshotUrl>001 http://www.bbc.co.uk/news/world-europe-33284937#sa-ns_mchannel=rss&ns_source=PublicRSS20-sa</snapshotUrl>
  <doFetch>true</doFetch>
  <doPopulate>true</doPopulate>
  <docType>item</docType>
  <repItemType>aspire/document</repItemType>
  <createdBy/>
  <connectorSpecific>
    <field name="comments"/>
    <field name="link">http://www.bbc.co.uk/news/world-europe-33284937#sa-ns_mchannel=rss&ns_source=PublicRSS20-sa</field>
    <field name="description">One man decapitated and several hurt in suspected Islamist attack on factory near Lyon, French sources say.</field>
    <field name="title">'Man decapitated' in French attack</field>
    <field name="uri">http://www.bbc.co.uk/news/world-europe-33284937</field>
  </connectorSpecific>
  <lastModified>2015-06-26T09:39:05Z</lastModified>
  <connectorSource type="rss">
    <feedUrls>
      <feedUrl>http://feeds.bbci.co.uk/news/rss.xml?edition=uk</feedUrl>
      <feedUrl>http://feeds.skynews.com/feeds/rss/uk.xml</feedUrl>
    </feedUrls>
    <indexContainers>false</indexContainers>
    <displayName>RSS Connector</displayName>
  </connectorSource>
  <action>add</action>
  <hierarchy>
    <item id="12FEEA65F246F754CD14CDE5AD5029D6" level="1" url="http://www.bbc.co.uk/news/world-europe-33284937#sa-ns_mchannel=rss&ns_source=PublicRSS20-sa"/>
  </hierarchy>
  <httpResponse code="200" source="FetchURLStage">OK</httpResponse>
  <protocol source="FetchURLStage/protocol">http</protocol>
  <host source="FetchURLStage/host">www.bbc.co.uk</host>
  <mimeType source="FetchURLStage/mimeType">text/html</mimeType>
  <encoding source="FetchURLStage/encoding">utf-8</encoding>
  <extension source="FetchURLStage">
    <field name="status">HTTP/1.1 200 OK</field>
    <field name="Server">Apache</field>
    <field name="Content-Type">text/html; charset=utf-8</field>
    <field name="X-News-Data-Centre">cwwtf</field>
    <field name="Content-Language">en-GB</field>
    <field name="X-PAL-Host">pal071.back.live.cwwtf.local:80</field>
    <field name="X-News-Cache-Id">97231</field>
    <field name="Content-Length">115141</field>
    <field name="Date">Fri, 26 Jun 2015 09:59:47 GMT</field>
    <field name="Connection">keep-alive</field>
    <field name="Set-Cookie">BBC-UID=9535987d82a209d365620e344160f72abcd7544074e4c1aeba21b4b40e9538d00Java/1.7.0_67; expires=Tue, 25-Jun-19 09:59:47 GMT; path=/; domain=.bbc.co.uk</field>
    <field name="Cache-Control">private, max-age=60, stale-while-revalidate</field>
    <field name="X-Cache-Action">HIT</field>
    <field name="X-Cache-Hits">54</field>
    <field name="X-Cache-Age">49</field>
    <field name="X-LB-NoCache">true</field>
    <field name="Vary">X-CDN,X-BBC-Edge-Cache,Accept-Encoding</field>
  </extension>
  <title source="ExtractTextStage/title">'Man decapitated' in French attack - BBC News</title>
  <contentType source="ExtractTextStage/Content-Type">text/html; charset=UTF-8</contentType>
  <description source="ExtractTextStage/description">One man decapitated and several hurt in suspected Islamist attack on factory near Lyon, French sources say.</description>
  <extension source="ExtractTextStage">
    <field name="robots">NOODP,NOYDIR</field>
    <field name="og:type">article</field>
    <field name="twitter:title">'Man decapitated' in French attack - BBC News</field>
    <field name="x-audience">Domestic</field>
    <field name="twitter:domain">www.bbc.co.uk</field>
    <field name="og:locale">en_GB</field>
    <field name="msapplication-TileImage">http://static.bbci.co.uk/news/1.75.0364/windows-eight-icon-144x144.png</field>
    <field name="x-country">gb</field>
    <field name="cleartype">on</field>
    <field name="og:article:author">BBC News</field>
    <field name="resourceName">http://www.bbc.co.uk/news/world-europe-33284937#sa-ns_mchannel=rss&ns_source=PublicRSS20-sa</field>
    <field name="CPS_AUDIENCE">Domestic</field>
    <field name="dc:title">'Man decapitated' in French attack - BBC News</field>
    <field name="viewport">width=device-width, initial-scale=1, user-scalable=1</field>
    <field name="twitter:creator">@BBCNews</field>
    <field name="theme-color">#bb1919</field>
    <field name="og:title">'Man decapitated' in French attack - BBC News</field>
    <field name="og:article:section">Europe</field>
    <field name="application-name">BBC News</field>
    <field name="mobile-web-app-capable">yes</field>
    <field name="og:description">One man decapitated and several hurt in suspected Islamist attack on factory near Lyon, French sources say.</field>
    <field name="X-UA-Compatible">IE=edge,chrome=1</field>
    <field name="apple-mobile-web-app-title">BBC News</field>
    <field name="twitter:card">summary_large_image</field>
    <field name="X-Parsed-By">org.apache.tika.parser.DefaultParser</field>
    <field name="og:site_name">BBC News</field>
    <field name="og:url">http://www.bbc.co.uk/news/world-europe-33284937</field>
    <field name="og:image">http://ichef.bbci.co.uk/news/1024/media/images/77623000/png/_77623460_breaking_image_large-3.png</field>
    <field name="msapplication-TileColor">#bb1919</field>
    <field name="Content-Location">http://www.bbc.co.uk/news/world-europe-33284937#sa-ns_mchannel=rss&ns_source=PublicRSS20-sa</field>
    <field name="Content-Encoding">UTF-8</field>
    <field name="twitter:description">One man decapitated and several hurt in suspected Islamist attack on factory near Lyon, French sources say.</field>
    <field name="twitter:site">@BBCNews</field>
    <field name="twitter:image:src">http://ichef.bbci.co.uk/news/560/media/images/77623000/png/_77623460_breaking_image_large-3.png</field>
  </extension>
  <content source="ExtractTextStage"><![CDATA[
 
    'Man decapitated' in French attack
        	    26 June 2015
	From the section Europe
        A man has been beheaded and at least one other person injured in a suspected Islamist attack on a factory near the French city of Lyon. 
Several small explosive devices were also set off at the Air Products factory in Saint-Quentin-Fallavier, sources said. 
The alleged attacker is said to have been carrying an Islamist flag, which was found nearby.
A man has been arrested, officials say.
Interior Minister Bernard Cazeneuve is  said to be on his way to the scene.

  Copyright © 2015 BBC. The BBC is not responsible for the content of external sites. Read about our approach to external linking.  
]]></content>
</doc>