An Exercise In Optimization

Sometimes when developing code you can spend to much time trying to optimize code that does not need to be optimized, especially if it makes the code harder to understand (often negatively referred to as "premature optimization").  However, sometimes optimization can be very worthwhile.

A quick example to share.  I was working on a segment of code last week that was taking a long time to run; on the order of 5+ minutes for something that seemed like it should be much quicker.  The code made sense when looked at, but it definitely felt as though this one might be worth spending time to optimize.

Upon closer inspection, I realized that a key index was missing from the database lookup.  This code has recently changed from processing a flat file to pulling information from a Sqlite database.  The table it was having to query was large, and the column it was querying against wasn't part of an index.  Upon adding the index, the run time of the code dropped from over 5 minutes down to 31 seconds.

All done!  Or not.  Once you are there, it's worth making sure you are not overlooking other possible issues.  I realized it was also querying one other table that, although it was smaller, was suffering from the same issue.  After adding an appropriate index for that table as well, the run time had dropped to 15 seconds.

But that's not all!  Upon a closer review of the code, I realized it was repeatedly looking up data with the same parameters in a lot of cases, so I added an in memory cache to cache that data as needed throughout the processing.  This dropped the total run time down to 5 seconds.

And then I realized I had also left out from the caching another lookup that was only needed when populating the cache.  After moving that last lookup to the code that utilized the cache, the run time had dropped again down to 14 milliseconds.

From 5+ minutes to 14 milliseconds.

I wasn't aware I could make it that fast, but by being mindful of what seems reasonable and being careful in my review of how the processing was operating (and not exclusively looking at just the database or just one segment of code), I was able to dramatically improve it.  Sometimes optimization can be very worthwhile.  Just use your best judgement on where it seems worthwhile, and where it would just complicate the code.

Handling EVE ESI Authentication with Spring Boot

Continuing on from my prior post on reading from the EVE ESI API with Java, today I provide some details on handing the EVE ESI authentication flow with Spring Boot.

With ESI, they expect your application to be a web application, whether it is one or not.  If it's not, things get more complicated.  The problem stems from their new flow where you are supposed to redirect to them for EVE sign on and then they redirect back to you with a code you use to obtain access to restricted data.  This is problematic if your application isn't a web application.

If you are using Java, this problem happens to be an ideal use for Spring Boot.  With Spring Boot, configuration is mostly a breeze and an application server like Tomcat can be easily embedded in your Spring Boot application to handle the redirect and callback.

First, a quick overview of the steps you need to take, then below I provide an example Controller class for a Spring Boot application that can handle all of this.

1. Determine your IP address, pick a port, choose a callback request mapping, and build the URL for your application from all that.  (example: http://[your IP]:[port]/[request mapping]).  It is recommended you use https, but doing so will require additional setup I'm not going into here.
2. Register your application at the EVE Developers Site, using the URL from the previous step as your application callback URL.
3. Access your router and forward TCP for your port to the machine that will be running your application.
4. Create your Spring Boot application.  Maven dependencies provided below the Controller example should you find it helpful.
5. Run your application.  Following the controller flow intended for the example, you would have an index page with a link to your /launchSignOn page.  Clicking that should redirect to EVE where you sign in.  Following that, it should redirect back to your application, where your application then takes the code sent back, makes a post request to get an access token (which includes a refresh token you will want), and makes a request to get character info, and then do whatever.  If you are just trying to get a refresh token and character ID for a different application, it can just display the fresh token and character ID on the screen for your to copy down.

Example Spring Boot Controller 

@Controller
public class SampleController {

 // these variables could come from a properties file or elsewhere, shown here as constants 
 // just for the sake of this example
 private static final String SCOPE = "esi-markets.read_character_orders.v1";  // example
 private static final String MY_CLIENT_URL = "?/eveCallback";  // callback URL for your application
 private static final String MY_CLIENT_ID = "?";   // client ID for your registered EVE application
 private static final String MY_SECRET_KEY = "?";  // secret key for your registered EVE application
 private static final String STATE = "?";            // your application state, if any
 private static final String MY_REFRESH_TOKEN = "?";  // saved after first sign in for subsequent use
 
 /**
  * A start page, should you want one.  It could have links to /launchSignOn and /testRefreshToken
  * 
  * @return view page reference
  */
 @RequestMapping(value="/", method=RequestMethod.GET)
 public String startPage() {
  return "index";
 }
 
 /**
  * Begin authentication process.  This will redirect to EVE sign on which will later redirect back 
  * to /eveCallback.  The redirect URL you here should be the same as the one you entered for your
  * registered EVE application, and it should point to your callback request mapping.
  * 
  * @return redirect URL to EVE sign on
  */
 @RequestMapping(value="/launchSignOn", method=RequestMethod.GET)
 public String launchSignOn() {
  return "redirect:https://login.eveonline.com/oauth/authorize"
    + "?response_type=code"
    + "&redirect_uri=" + MY_CLIENT_URL
    + "&client_id=" + MY_CLIENT_ID
    + "&scope=" + SCOPE
    + "&state=" + STATE;
 }
 
 /**
  * EVE sign on should redirect back to here. Now you can send a post request to get an
  * access token (which also includes a refresh token you probably will want to save), and if needed,
  * you can make a request to obtain character details as well (in particular, you will probably
  * want the character ID).
  *  
  * @param model
  * @param code    the code you need to request an access token
  * @param state   your appliction state info, should match what you originally sent in the redirect
  * 
  * @return view page reference, which could display info you want to see about the token or character.
  */
 @RequestMapping(value="/eveCallback", method=RequestMethod.GET)
 public String catchCallback(ModelMap model, @RequestParam String code, @RequestParam String state) {
  // do something with your state if needed
  MoxyJsonConfig moxyJsonConfig = new MoxyJsonConfig();
  ContextResolver<MoxyJsonConfig> jsonConfigResolver = moxyJsonConfig.resolver();
  Feature basicAuth = HttpAuthenticationFeature.basic(MY_CLIENT_ID, MY_SECRET_KEY);
  Client client = ClientBuilder.newBuilder()
    .register(basicAuth)
    .register(jsonConfigResolver)
    .build();
  MultivaluedMap<String, String> formData = new MultivaluedHashMap<String, String>();
  formData.add("grant_type", "authorization_code");
  formData.add("code", code);
  AccessToken accessToken = client.target("https://login.eveonline.com/oauth/token")
    .request(MediaType.APPLICATION_JSON_TYPE)
    .post(Entity.form(formData), new GenericType<AccessToken>(){});
  Feature bearerAuth = OAuth2ClientSupport.feature(accessToken.getAccess_token());
  client = ClientBuilder.newBuilder()
    .register(bearerAuth)
    .register(jsonConfigResolver)
    .build();
  EveCharacter character = client.target("https://login.eveonline.com/oauth/verify")
    .request(MediaType.APPLICATION_JSON_TYPE)
    .get(new GenericType<EveCharacter>(){});
  model.put("accessToken", accessToken);
  model.put("character", character);
  return "showTokenAndCharacterId";
 }
 
 /**
  * Uses a previously stored refresh token to obtain a new access token and character information.
  * 
  * @param model
  * 
  * @return view page reference, could display whatever info you want to see about token or character.
  */
 @RequestMapping(value="/testRefreshToken", method=RequestMethod.GET)
 public String testRefreshToken(ModelMap model) {
  final MoxyJsonConfig moxyJsonConfig = new MoxyJsonConfig();
  final ContextResolver<MoxyJsonConfig> jsonConfigResolver = moxyJsonConfig.resolver();
  Feature basicAuth = HttpAuthenticationFeature.basic(MY_CLIENT_ID, MY_SECRET_KEY);
  Client client = ClientBuilder.newBuilder()
    .register(basicAuth)
    .register(jsonConfigResolver)
    .build();
  MultivaluedMap<String, String> formData = new MultivaluedHashMap<String, String>();  
  formData.add("grant_type", "refresh_token");
  formData.add("refresh_token", MY_REFRESH_TOKEN);
  AccessToken accessToken = client.target("https://login.eveonline.com/oauth/token")
    .request(MediaType.APPLICATION_JSON_TYPE)
    .post(Entity.form(formData), new GenericType<AccessToken>(){});
  Feature bearerAuth = OAuth2ClientSupport.feature(accessToken.getAccess_token());
  client = ClientBuilder.newBuilder()
    .register(bearerAuth)
    .register(jsonConfigResolver)
    .build();
  EveCharacter character = client.target("https://login.eveonline.com/oauth/verify")
    .request(MediaType.APPLICATION_JSON_TYPE)
    .get(new GenericType<EveCharacter>(){});
  model.put("accessToken", accessToken);
  model.put("character", character);
  return "showTokenAndCharacterId";
 }
}

Example Maven Dependencies

  <parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.0.1.RELEASE</version>
  </parent>
  <dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
     <groupId>org.springframework.boot</groupId>
     <artifactId>spring-boot-starter-tomcat</artifactId>
    </dependency>
    <dependency>
     <groupId>org.springframework</groupId>
     <artifactId>spring-webmvc</artifactId>
    </dependency>
    <dependency>
     <groupId>org.apache.tomcat.embed</groupId>
     <artifactId>tomcat-embed-jasper</artifactId> <!-- only needed if using JSP -->
     <scope>provided</scope>
    </dependency>
    <dependency>
     <groupId>org.glassfish.jersey.core</groupId>
     <artifactId>jersey-client</artifactId>
    </dependency>
    <dependency>
     <groupId>org.glassfish.jersey.inject</groupId>
     <artifactId>jersey-hk2</artifactId>
     <version>2.26</version>
    </dependency>
    <dependency>
     <groupId>org.glassfish.jersey.media</groupId>
     <artifactId>jersey-media-moxy</artifactId>
     <version>2.26</version>
    </dependency>
    <dependency>
     <groupId>org.glassfish.jersey.security</groupId>
     <artifactId>oauth2-client</artifactId>
     <version>2.26</version>
    </dependency>
  </dependencies>

Calling EVE OpenAPI with Java

If you are into Java and want a quick example of a way you can read data from the EVE OpenAPI, here is an example I wrote using Jersey 2.26.

import java.util.List;

import javax.ws.rs.client.Client;
import javax.ws.rs.client.ClientBuilder;
import javax.ws.rs.core.GenericType;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.ext.ContextResolver;

import org.glassfish.jersey.moxy.json.MoxyJsonConfig;

/**
 * Example of calling the EVE OpenAPI using Jersey 2.26.  Specific Jersey components needed
 * include jersey-client 2.26, jersey-hk2 2.26, and jersey-media-moxy 2.26.
 */
public class EveOpenAPIExample {

 // variable names need to match names from the EVE OpenAPI
 private static class Region {
  private String region_id;
  private String name;
  private List<String> constellations;
  private String description;
  public String getRegion_id() {
   return region_id;
  }
  public void setRegion_id(String region_id) {
   this.region_id = region_id;
  }
  public String getName() {
   return name;
  }
  public void setName(String name) {
   this.name = name;
  }
  public List<String> getConstellations() {
   return constellations;
  }
  public void setConstellations(List<String> constellations) {
   this.constellations = constellations;
  }
  public String getDescription() {
   return description;
  }
  public void setDescription(String description) {
   this.description = description;
  }
 }
 
 public static void main(String[] args) {

  // setup the service information we will need for the example
  final String target = "https://esi.tech.ccp.is/latest/";
  final String path = "universe/regions/{regionid}/";
  final String regionId = "10000002";
  final String dataSource = "tranquility";
  
  // create the client configured to use MOXy for JSON binding
  final MoxyJsonConfig moxyJsonConfig = new MoxyJsonConfig();
  final ContextResolver jsonConfigResolver = moxyJsonConfig.resolver();
  Client client = ClientBuilder.newBuilder()
   .register(jsonConfigResolver).build();
  
  // execute example call on EVE OpenAPI to read information on a region
  Region region = client.target(target)
   .path(path)
   .resolveTemplate("regionid", regionId)
   .queryParam("datasource", dataSource)
   .request(MediaType.APPLICATION_JSON_TYPE)
   .get(new GenericType<Region>(){}); 
  
  // utilize the data
  System.out.println("Read information for region " + region.getName());
  
  System.exit(0);
 }
}

Tools Review!

It's time to take a look at what arguably might be my most interesting and useful blog posts.  The tools!  No, not 3rd party tools that exist that I maybe provide links to; I'm talking about the ones here on this blog, most of which are driven by JavaScript on the back end.  All of them are really just at the proof of concept level, and I think it's time to consider which ones, if any, should be more fully developed.

Here are the tools I've created in the past, and why:

• Eve To Local Time Tool -- many a time, I've seen an event come up in Eve - perhaps a reinforcement timer expiring or a scheduled fleet of some kind - and then people start trying to figure out what time that means for them in whatever local time they are in.  A lot of folks, myself included, would probably find it convenient to copy the date/time from Eve into a converter that will convert it to your local time.  That's what this tool does.
• Invention Calculator -- Other invention calculators exist on the Internet, but at the time of this ones writing, the others were all outdated and didn't reflect the most recent changes from CCP.  This tool tells you your chance at a successful invention given your skills and components.
• Ship Scanning Tool -- When scanning a ship to determine it's fit, you don't really know which fittings will be returned.  A scan usually doesn't show you all of them, but some seemingly random subset of them.  This tool provides a way to copy and track repeated scans.  It keeps a running list of everything scanned so far, and whether it represents the complete fit for the ship being scanned.
• Missile Damage Calculation Tool -- This tool attempts to calculate what percentage of missile damage will actually be applied to an enemy ship based on skills, fittings, missile type, and opponent ship type.  Even better, it provides a graph for the damage percentage applied over a range of opponents ship speeds from 0 m/s up to the ships typical maximum speed.

Personally, I was thinking of further developing the Missile Damage Calculation Tool.  Currently, it only has back-end data for a few PVE ships, and only for a couple of missile types.  I was thinking of making it far more useful in general and for PVP, by adding player ship types to the ship list, adding in all the remaining missile types, and adding new inputs for the new new missile guidance modules.

Which of these would you like to see more fully developed, and in what ways?  Do you have ideas for other tools that I might consider developing?  Feel free to share your ideas.

Eve To Local Time Tool

One annoying task every Eve player faces is having to figure out what the local time will be for some event in Eve. Provided here is a quick, simple tool for converting Eve times to local times.

Enter Eve time in format "yyyy.MM.dd HH:mm"

Eve Time:

Local time: